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Abstract 

The study evaluated the training effectiveness of an intelligent tutoring system 
(ITS) for the Remote Manipulator System (RMS). The study examined how well 
individuals learn the training content and skills from the RMS ITS and to what 
extent the content and skills learned using the ITS transfer to RMS task 
performance in the SES, a high fidelity simulator. Three astronauts completed 
8 2-hour ITS sessions addressing movement in three coordinate systems , grapple, 
ungrapple, berth, and unberth procedures, and singularities and reach limits. 
Their performance was also observed in a SES training session. Performance data 
was collected using multiple measures: ITS task performance, transfer 

performance on the SES, a conceptual knowledge test, an opinion survey completed 
by astronauts, and comments and ob serva tions from astronauts and trainers. 
Results indicated the RMS ITS to be moderately effective and provided evidence 
of the efficacy of ITS's, in general. Comments and suggestions are provided 
relating to how the ITS could be improved and to enable decision makers to judge 
the effectiveness of the RMS ITS. 
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An Evaluation of Training Effectiveness 
of an Intelligent Tutoring System 


Education and training in industry is time-consuming and expensive. As 
tools and equipment become more sophisticated, training costs (both time and 
money) will increase also. Indeed, many of the tasks performed by NASA 
personnel, e.g., mission specialists or mission controllers, require extensive 
training and elaborate simulation equipment. Tools are needed to reduce the high 
training costs and time requirements on complex tasks. They are also needed to 
facilitate training in situations requiring sophisticated simulation equipment 
and where too few personnel are available to conduct the training. 

Intelligent Tutoring Systems (ITS's) offer a means for addressing these 
training needs. ITS's have already been developed to teach a variey of topics 
and task activities in education al sett ings and to a lesser extent in industry 
settings (Wenger, 1987). Moreover, ITS's have been recently developed at NASA 
to provide training on specific tasks (e.g.. Payload-assist module Deploys, RMS 
use) and a general architecture has been proposed to reduce the costs and time 
required to build ITS's for other tasks (Loftin, Wang, Baffes, & Hua, 1988). 

As a result of building new ITS's, much attention has been given to the 
design issues of ITS's (based on Loftin's general architecture [Loftin et al., 
1988]). However, little attention has been given to evaluation of ITS's. 
Evaluation is thought to be time consumi ng and costly (Frye, Littman, & Soloway, 

1987) . Moreover, there are not clear guidelines for how to assess a system's 
effectiveness (Burns & Capps, 1988). Traditional training evaluation tools and 
procedures (e.g., Goldstein, 1986) offer a starting point but may not be 
sufficient to evaluate ITS's (Goldstein, 1989). 

Traditionally, evaluation has been discussed in terms of two categories: 
formative and summative (Scriven, 1967 ). Formative evaluation investigates 
whether the program is operating as planned or if improvements are needed before 
a program is implemented. Summative evaluation examines the effectiveness of the 
final product. Given the newness and exploratory nature of many ITS's, the focus 
of evaluation of ITS's should be more on formative evaluation (Littman & Soloway, 

1988) . Indeed, of the few evaluation efforts reported to date, most have been 
formative. However, most evaluatio ns h av e also been relatively informal (Littman 
& Soloway, 1988). Further, some evaluation efforts have been primarily 
descriptive or qualitative in nature (see Wenger, 1987, p. 59, p. 133 for 
examples) or have used weak experimental designs such as pre-test and post-test 
with no control groups (see Wenger, 1907, p. 96). A few more rigorous 
evaluations have been completed, but primarily in the lab or in educational 
settings (e.g., Anderson, Boyle, & Reiser, 1985). Few evaluation efforts have 
been completed in industrial settings, although more attention is beginning to 
be focused on this (Govindaraj, 1988). 

Evaluation efforts to date are also likely to be small in number and more 
informal because guidelines are not yet well developed for evaluating ITS's. One 
set of guidelines offered for performing formative evaluations focuses on 
external and internal evaluation (Littman and Soloway, 1988). Internal 
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evaluation focuses on the system's architecture (which is beyond the scope of the 
current project). External evaluation focuses on how the system affects 
students ' problem solving proces ses. The rationale underlying external 
evaluation is that using a more traditional evaluation approach of assessing 
correct versus incorrect performance is not of sufficient detail. To more 
properly evaluate the system, one also needs to examine the reasons underlying 
students' errors. Information on the types of errors students make can then be 
used to improve the remediation provided to students. The interest in using a 
more fine-grained analysis of students' performance reflects the purpose of 
formative evaluation: to improve an ITS in development. The coarser, but still 
important, measures relating to correct versus incorrect performance reflect a 
focus on summative evaluation. 


Purpose 

The purpose of the current research project was to evaluate the training 
effectiveness of an ITS developed for training RMS tasks. It was not possible 
to examine the ITS in relation to other training interventions. Thus, the 
relative effectiveness of the RMS ITS compared to other forms of training was 
beyond the scope of this study and remains a question for furture research. 
Rather, this study focused on examining comprehensive information related to the 
RMS ITS. 

The project extended previous evaluation research in two ways. First, the 
evaluation attempted to apply a set of guidelines proposed (Littman and Soloway, 
1988) for conducting an external, formative evaluation. Previous work by the 
author examining other ITS's at NASA also had a more formative focus although it 
relied more on coarser measures, e.g., counts of subtasks completed or number of 
errors (Johnson, 1989, 1990). In addition, previous work by the author reported 
some descriptive or qualitative information, e.g., ITS user impressions. Thus, 
this earlier work provided a foundation for expanding evaluation techniques to 
use more fine-grained measures. 

The current study attempted to collect more fine-grained data using a two- 
part approach. More specifically, we used a variety of measures of learning and 
performance, including assessing ITS task performance, transfer of training 
performance on the SES, a conceptual knowledge test, a survey of opinions 
collected from the astronauts, as well as comments and observations from the 
astronauts and task experts, i.e., the trainers. Moreover, multiple, fine- 
grained dimensions of performance were assessed on the ITS and SES, including 
number of trials failed, time required per trial, accuracy, efficiency, camera 
use, etc. 

This two-part approach offered the opportunity to examine detailed 
information on knowledge and performance and provided an opportunity to draw 
stronger conclusions regarding the results to the extent that data collected 
using different measures converged. This also enabled the researcher to state 
some conclusions although data was only available for a few subjects. Given the 
demands placed on astronauts, access to large numbers of subjects was not 
possible. Thus, more comprehensive data was needed to enable decisions regarding 
ITS effectiveness, especially data collected using diverse measures. 
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This approach was consistent with Kirkpatrick (1977) who described four 
levels of evaluation: reactions, learning, performance, and effectiveness. 
Reaction criteria refer to individuals Z thoughts and feelings about the program. 
This data was collected in the current study through information comments and an 
opinion survey. Learning criteria are more rigorous and address skill mastery 
or assessment of concept understanding. This could be called the validity of 
training. in the current study we collected objective performance data on the 
RMS ITS and administered a conceptual knowledge test. Kirkpatrick's performance 
criteria refer to the extent to which skills and knowledge learned during 
training transfer to job performance. This was referred to as transfer of 
training in the current study and was assessed by examining performance on the 
SES. Finally, effectiveness cr iteria r efer to measures of training results in 
the organization which was not directly examined in the current study. 

Second, the ITS evaluation combined aspects of both formative and summative 
evaluation. Although a formative evaluation approach was more appropriate given 
the newness of the ITS, aspects of a summative evaluation, e.g., performance 
accuracy or time requirements, could aid decision makers in making initial 
judgments of the effectiveness of the RMS ITS as a training tool and of the 
efficacy of iTS's in general. 


Method 


Subjects 

Three male astronauts who were scheduled to begin Remote Manipulator System 
(RMS) 2000 level training participated in a modified training program 
incorporating an RMS ITS. The RMS is a robotic arm used to deploy and/or 
retrieve shuttle payloads (e.g., satellites). The astronauts were informed that 
the training they would be receiving was a modified version of the traditional 
level 2000 training and were given a description of the modified training flow. 
Their participation was voluntary, and informed consent was obtained. The 
astronauts were also debriefed at the end of the project. 

Training Modifications and Research Project Procedure 

The iTS-modified RMS training affected the content of three RMS 2 000 level 
training modules: PDRS OPS 212 4, PDRS NOM OPS 2124, and PDRS NOM OPS 2115. 
These three modules usually require a total of 8.5 hours of training time. 
However, they required a total of 4.5 hours in the modified training curriculum 
because training content which replicated ITS training content was removed. In 
addition, performance information w as collected during PDRS NOM OPS 2215 (3.5 
hours). Subjects completed RMS 2000 level training modules through MDF FAM 2324 
with no modifications. They then completed eight, 2-hour RMS ITS training 
sessions in parallel with the three modified modules (PDRS OPS 2124, PDRS NOM OPS 
2124, and PDRS NOM OPS 2115) . The RMS 2000 level training sequence was increased 
by 12 hours to incorporate the ITS. As part of an early RMS ITS training 
session, subjects received an one-hour briefing RMS coordinate systems from a 
Shuttle trainer, subjects 1 and 3 received an 1-hour briefing during ITS Session 
4; Subject 2 received a 2-hour briefing on both coordinate systems and the PDRS 
Overview information during ITS Sesiion 1. Also, subjects had access to the 
"Payload Deployment and Retrieval System Overview Workbook" (PDRS OV 2102) used 
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in Shuttle RMS training. Performance data was collected during the PDRS NOM OPS 
2215 training module on the SES; this module was unmodified except for the 
observation and recording of subject performance on a subset of the tasks 
performed. Subjects then returned to the original training flow, receiving the 
same subsequent RMS training as all other astronauts. 

ITS Tasks: Description and Performance Measures 

The RMS 2000 level training was supplemented with lessons from the P2T2 RMS 
ITS developed by Global Information Systems Technology (NASA P2T2 Intelligent 
Trainer Final Report, 1991). The ITS overlaid training content on the P2T2, an 
existing kinematic simulator of the shuttle's robotic arm. The ITS -modified 
training included ITS lessons on the Orbiter Unloaded, Orbiter Loaded, and End 
Effector coordinate systems as well as ITS lessons on Grappling, Ungrappling, 
Berthing, Unberthing, Recognizing Singularities, visualizing Singularities, 
Recognizing Reach Limits, and Visualizing Reach Limits. These lessons were a 
subset of the part tasks (i.e., subtasks of deploys or retrievals) available on 
the ITS. The ITS also provided part tasks on the Payload coordinate system and 
Loaded and Unloaded Arm Phasing which were not used in the current study. The 
Payload part task was not used due to a 180 degree reversal in one of the axes 
included in the part task. This reversal was discovered by a trainer after the 
project began. The task could have been changed to reverse the axis but it was 
decided that experience with the oth er three coordinate systems would be 
sufficient for the evaluation of the ITS, especially given the time constraints 
of the astronauts. The Loaded and Un loa ded Arm Phasing part tasks were not used 
because they used concepts addressed in earlier part tasks. Finally, two whole 
tasks — Retrieval and Deploy tasks — were available on the ITS but were not used 
for this project due to time constraints. 

To perform ITS lessons, subjects used translational and rotational hand 
controls, a keyboard, and a control box to manipulate task components viewed on 
a computer monitor. The left hand control, the translator, enabled movement of 
the RMS on the x, Y, and Z axes with the orientation of the axes dependent on the 
coordinate system being used. The right hand control enabled rotation of the RMS 
on the X, Y, and Z axes, with the orientation again dependent on the coordinate 
system being used. The keyboard was used to enter information related to the 
payload. The control box enabled su bjects to change settings on the control 
panel viewed on the computer monitor. 


The computer monitor displayed four windows. The lower left window 
displayed the control panel which was^ accessed through the control box. The 
control panel enabled the control of various RMS operations, e.g., the selection 
of the appropriate Mode — Unloaded, Loaded, etc. The upper left and upper right 
windows offered views of the RMS and shuttle bay, with the view dependent on 
camera selection and orientation. The control box enabled one to manipulate 
camera views in these two windows. Finally, the lower right window provided task 
status information and ITS task controls which were accessed using a mouse (e.g., 
exit the ITS, go on to the next task) . For more complete information on displays 
and ITS usage, see the P2T2 Intelligent Trainer Final Report (Global Information 
Systems Technology, 1991). 
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Coordinate System Tasks. Subjects completed ITS part tasks relating to the 
orbiter Unloaded, Loaded, and End Effector coordinate systems. Part tasks are 
subtasks of the deploy and retrieval tasks. The part tasks relating to 
coordinate systems aided subjects in visualizing and moving the RMS. For each 
coordinate system, subjects first performed a set of translation tasks. The 
translation tasks had four levels of complexity (LOC's): movement in one, two, 
then three dimensions, and finally movement of greater distance in three 
dimensions and without a ghost arm (indicating the target position). Subjects 
next performed a set of rotation tasks, completing the same four levels of 
complexity. Then, subjects completed a set of integrated tasks requiring both 
translation and rotation, again at four levels of complexity. Subjects completed 
the translation, rotation, and integrated hand control tasks for the Unloaded, 
Loaded, and finally the End Effector coordinate systems. 

Within each level of complexity, subjects performed 2 to 5 trials. If 
subjects passed the first two trials, they were advanced to the next level of 
complexity. otherwise, subjects were required to pass 5 successive trials to 
advance to the next level of complexity. Generally, if the subject failed a 
trial, s/he was required to reattempt the 5 successive trials required to pass 
to the next level of complexity. However, if the subject passed 3 or 4 trials, 
then failed a trial but passed the next, the subject was given another chance 
before being required to reattempt the 5 successive trials required to advance. 
This was true of both the coordinate. systems and the procedural part tasks. 

Performance on the coordinate system part tasks was assessed in terms of 
accuracy and efficiency. Accuracy accounted for 75% of the overall score and 
efficiency for 25%. Subjects were required to attain at least 75 total points 
(out of 1 0 0 ) to pass the trial. The default criteria levels provided by the RMS 
ITS were used (Global information Systems Technology, 1991). That is, accuracy 
referred to the distance from the target coordinates upon task completion. The 
translation allowance was 5 inches. The rotation allowance was 5 degrees for 
roll and 8 degrees for pitch and yaw. Efficiency referred to the path and time, 
with path accounting for 80% of the efficiency score. The minimum passing score 
(a score of 75) for path was 1.5 times the minimum distance or rotation, i.e., 
no more than 50% farther than the minimum distance or rotation possible. For the 
time criteria, the minimum passing score (75) was obtained if one used the time 
allowed. A subject earned the maximum score on time (100) if s/he performed the 
task in half the allowed time. 

Procedural Tasks. subjects next completed a set of part tasks involving 
procedures. The procedural part tasks required the performance of sequences of 
actions based on the PDRS Operations Checklists P2T2 Flight Supplement Generic 
Procedures (see Appendix A). The Generic Procedures were developed by the space 
Shuttle and space Station Programs , sub jects were provided with a copy of the 
Generic Procedures and referred to this document as they performed the Grapple, 
Ungrapple, Berth, Unberth, Recognizing Singularities, Visualizing Singularities, 
Recognizing Reach Limits, and visualizing Reach Limits part tasks. 

The procedural tasks are categorized into two groups for ease of description 
below. The groups were determined by the similarity of the scoring procedures. 
The Grapple, Ungrapple, Berth, and Unberth tasks will be described first. For 
the Grapple task, the RMS started in pre-grapple position and subjects completed 
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the grapple procedure (see Generic Procedures in Appendix A). There were three 
levels of complexity* In LOC 1 the camera views were preset for subjects. In 
LOC 2, subjects were required to adjust camera views. In LOC 3, subjects were 
required both to adjust camera views and grapple payloads with fixtures in 
different locations (e.g., top, side). The Ungrapple task involved the ungrapple 
or release procedures and movement to the pre-grapple position. In LOC 1, 
subjects completed the Release procedure using preset camera views. In LOC 2, 
subjects completed the Grapple procedure and were required to adjust the camera 
views. In LOC 3, subjects alternately completed the Ungrapple and Release 
procedures, again adjusting camera views. The Berth task involved berthing the 
payload starting from the "low hover" position, i.e., from a position directly 
over the v-guides. LOC 1 offered subjects preset camera views and visual cues 
on the monitor. LOC 2 and 3 required the adjustment of camera views, and LOC 3 
also varied the position of the grapple fixture. The Unberth task involved 
unberthing the payload and moving it to "low hover" position. The LOC's were the 
same as for the Berth task. 

Within each LOC, subjects were required to pass 3 successive trials to 
advance to the next level, subjects' performance was again based on accuracy and 
efficiency; however, subjects were also evaluated on safety, correctness of 
procedure, and camera use. Accuracy referred to acceptable distances from the 
RMS to the target position and attitude for the task. The translation allowance 
was 8 inches for the grapple and berth tasks; the rotation allowance was 2 
degrees for the berth task. Other allowan ces were the same as for the coordinate 
system tasks. Efficiency referred to path and time. Fly-to's, i.e., movement 
of the RMS, during the task were required to be at most 1.76 times the minimum 
translation and rotation to pass the trial. Simultaneous hand controller use was 
required to occur at least 50% of the time possible to pass. The time criteria 
were the same as for the coordinate system tasks. Safety referred to preventing 
contact of the RMS with any other structure. Points were deducted for each 
collision (25 points), movement in coarse rate when one should be in vernier rate 
(4 points), entering direct mode (10 points), and movement into a reach limit or 
singularity (10 points). Procedure referred to whether the correct sequence of 
actions was completed. Twenty points was deducted for each procedure error. 
Camera Use referred the percentage of time the cameras were used correctly. 


The second group of procedural part tasks included Recognizing 
Singularities, Visualizing Singularities, Recognizing Reach Limits, and 
Visualizing Reach Limits. For the Recognizing Singularities task, subjects 
identified shoulder yaw, wrist yaw, and planar pitch singularities in LOC's 1, 
2, and 3, respectively, and singularities of any type in LOC 4. For the 
Visualizing Singularities tasks, subjects identified the type of singularity 
likely to occur from given RMS configurations, identified the type of input 
required to drive the RMS into singularity, drove the RMS into the singularity, 
and resolve the singularity, adjusting camera views as necessary. In LOC 1 
singularities were obvious and preset camera views were used. LOC 2 required 
camera view adjustment. LOC 3 used, less obvious singularities and required 
camera view adjustment. For the Recognizing Reach Limits task, subjects 
identified 6 reach limits with the 6 LOC's reflecting the shoulder yaw, shoulder 
pitch, elbow pitch, wrist pitch, wrist yaw, wrist roll reach limits, 
respectively. For the visualizing Reach Limits tasks, subjects performed the 
same four subtasks as for the visualizing Singularities task. 
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Within each LOC, subjects were required to pass 3 consecutive trials to 
advance out of the Recognizing tasks and 5 consecutive trials to advance out of 
the Visualizing tasks, subjects' performance on the Recognizing tasks was based 
on Accuracy , i.e., correct/incorrect identifications • Subjects' performance on 
the Visualizing tasks was also evaluated on Accuracy ( correct /incorrect ) for the 
identification subtasks. Points were deducted for each incorrect identification 
after the first attempt for singularities (25 points) and reach limits (20 
points); 5 points were deducted for each initial incorrect hand controller choice 
for either task; 15 points were deducted for each additional incorrect hand 
controller choice. The Accuracy score was also reduced for hitting a reach limit 
(in singularity task) (50% reduction) , hitting a software stop (75%) , and hitting 
a hardware stop (100%). In addition, 35 points were lost for failure to drive 
into a reach limit, 20 points for failure to drive RMS into a singularity, 20 
points for failure to resolve a singularity, and 20 points for failure to drive 
into a target RMS configuration. Camera use and Efficiency were also assessed 
as described above for other procedural tasks. 

Survey of Content Knowledge 

Subjects completed a written survey of content knowledge, addressing their 
understanding of coordinate systems, sequences of RMS task activities, and 
singularities and reach limits upon completion of all ITS lessons (see Appendix 
B) . The survey was administered immediately prior to the SES lesson in which 
performance was observed. Thus, the test was administered between three and four 
weeks following the last ITS lesson. 

Transfer Performance Measures 


Following the ITS lessons, subjects completed the PDRS NOM OPS 2215 RMS 20 00 
level training module. This module involved performance on the SES of a variety 
of RMS tasks. Subjects' performance was observed on 5 tasks during the SES 
training session: fly-to positions/attitudes, grapple, ungrapple, berth, and 
unberth procedures. Four raters (one experimenter and three trainers) assessed 
subjects' performance on dimensions of these tasks (see Appendix C) . 

Qualitative Data 

Qualitative data was also collected both from astronauts and trainers. 
Astronauts completed an opinion survey relating to the ITS (see Appendix D), 
addressing their reactions to, and suggestions for modifications of the ITS. 
Further, their informal comments were solicited and recorded during the ITS 
lessons and during the PDRS NOM OPS 2215 training module. 

In addition, comments were solicited and recorded from two trainers who 
possessed Space Shuttle and/or Space Station training experience. Comments were 
solicited informally throughout the project and in a formal meeting after the 
completion of other data collection. 
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Coordinate System Tasks 

Unloaded Part Tasks, Mean performance data across the three subjects for 
the Unloaded part tasks are shown in Table 1. Two types of information are 
presented. Within each LOC, each subject performed at least two task trials in 
order to advance to the next LO C . Thu s , for each LOC, subjects' performance 
averaged across the task trials required to advance to the next LOC .is presented. 
In addition, performance data averaged across subjects' single best task trials 
within each LOC are presented. 

As shown, subjects required approximately 8 trials to pass the THC tasks, 
2 trials to pass the RHC LOC 1, 3, and 4 tasks, and between 2 and 5 trials to 
pass the Integrated LOC 1, 3, and 4 tasks. The 28 trial mean in THC LOC 3 was 
due to one subject who required 50 trials to pass this task. Also of interest 
was the apparent difficulty of learning tasks requiring movement in 2 dimensions 
(LOC 2). For the RHC and Integrated tasks, subjects found learning 2 -dimensional 
movement the most difficult. M ovement in 1 dimension presented few problems* 
Further, once movement in 2 -dimensions was mastered, movement in 3 dimensions 
(LOC 3) or across longer distances without a ghost arm (LOC 4) presented 
relatively difficulty. 

As expected, the time required to perform the tasks increased with the 
level of complexity involved (LOC 1 through 4). Also, subjects focused more on 
accuracy than on efficiency. Subjects were told that accuracy was more important 
than efficiency, and accuracy was a more important component of the total score. 
The lower efficiency scores reflect the greater emphasis on accuracy. 

Loaded Part Tasks . Mean performance data for the Loaded part tasks are 
shown in Table 2. Performance of the Loaded part tasks appeared to be easier for 
subjects, with subjects requiring approximately 2 trials to advance to the next 
LOC. This was expected given the similarity to the Unloaded part tasks. Only 
the Point of Resolution (POR; the origin of the coordinate system) differed for 
the two tasks, with the POR in the tip of the end effector for the Unloaded tasks 
and at a point in the payload (e.g., 6 inches from the tip of the end effector) 
for the Loaded tasks. 

Subjects required somewhat more time on average to perform the RHC tasks in 
the Loaded mode (between 32.7 and 202.7 sec.) than in the Unloaded mode (between 
23.7 and 103.2 sec.). Also, subjects again focused more on accuracy than on 
efficiency, receiving substantially higher accuracy scores. 

End Effector Part Tasks. Mean performance data for the End Effector part 
tasks are shown in Table 3. All three subjects performed THC LOC 1 through 4 
tasks for the End Effector mode. However, no subject passed LOC 4. Further, due 
to time constraints and subject preference, only one subject performed the RHC 
(only LOC 4) and Integrated (only LOC 1 through 3) part tasks. 

Performance of the End Effector THC part tasks appeared to be more difficult 
for subjects than the Loaded THC part tasks, requiring between 4.7 and 9.3 trials 
to pass LOC 1 through 3. In LOC 4, one subject performed 5 trials and a second 
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subject performed 27 trials without passing the task (i.e., passing 5 successive 
trials). Only one subject performed any other End Effector tasks. This subject 
required 2 trials to pass RHC LOC 4 and Integrated LOC 1 and 2, but 13 trials to 
pass Integrated LOC 3. 

However, the time required to perform task trials was similar to the 
Unloaded tasks, ranging from 22.7 to 146.4 sec. for the THC, RHC, and Integrated 
tasks. Also, similar to the Unloaded and Loaded part tasks, subjects focused 
more on accuracy than on efficiency and received lower efficiency scores. 


Table 1. Mean Performance for Unloaded Part Tasks. 


Variable 

THC 
LOC 1 

THC 
LOC 2 

THC 
LOC 3 

THC 
LOC 4 

RHC 
LOC 1 

RHC 
LOC 2 

RHC 
LOC 3 

RHC 
LOC 4 

# of Trials 
Required to 
Pass 

8.3 

8.3 

28.0 

7 . 0 

| 2.0 

j 

9.0 

2.0 

2.0 

# of Trials 
Failed 

2.3 

3.7 

8.7 

2.7 

1 0.0 

2.7 

0.0 

0.0 

Average Time 
per Trial 
( sec . ) 

26.3 

35.5 

38,8 

68.2 

1 27.3 

88.3 

80.7 

103.2 

Best Trial Time 
(Sec . ) 

17.7 

18.0 

32.0 

50.0 

| 25.7 

50.0 

71.3 

70.3 

Average 
Total Score 

89.9 

82.6 

80.4 

80.7 

| 94.8 

| 

j 

86.3 

92.9 

91.1 

Best Trial 
Total Score 

95.7 

92.9 

90 .3 

94.0 

| 98.2 

95.5 

94.7 

94.6 

Average 

Accuracy Score 

94.5 

87.0 

82.9 

87.2 j 

98.5 

96.4 

98.3 

98.2 

Best Trial 
Accuracy Score 

97.6 

92.5 

94,0 

96.7 j 

99.7 

98.6 

98.9 

98.0 

Average 

Efficiency 

Score 

76.0 

69.3 

72,7 

61.3 | 

] 

83.9 

! 

56.0 

76.6 

69.6 

Best Trial 
Efficiency 
Score 

90.1 

94.1 

79,3 

85.9 | 

93.7 

86.2 

82.1 

84.2 
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Table 2. Mean Performance for Loaded Part Tasks, 


Variable 

THC 
LOC 1 

THC 
LOC 2 

THC 
LOC 3 

THC 
LOC 4 

RHC 
LOC 1 

RHC 
LOC 2 

RHC 
LOC 3 

RHC 
LOC 4 

# of Trials 
Required to 
Pass 

2 . 0 1 

2 . 0 1 

2.0 

2.0 

2.7 

! 

! 

2.3 

2.7 

3.7 

# of Trials 
Failed 

0.0 

0.0 

0,0 

0.0 

1 0.0 

0.0 

0.3 

0.3 

Average Time 
per Trial 
( Sec . ) 

29.5 

50.0 

54.7 

68.2 

1 61.4 

229.7 

200.4 

169.4 

Best Trial Time 
(Sec . ) 

23.0 

37.0 

5477 

61.0 

| 32.7 

| 

175.7 

202.7 

152.3 

Average 
Total Score 

97.8 

93.3 

92.0 

94.0 

j 96.5 

i 

85.3 

84.5 

88.8 

Best Trial 
Total Score 

99.3 

98.1 

92.6 

95.4 

j 99.5 

89.9 

94.3 

94.7 

Average 

Accuracy Score 

97.6 

96.5 

94.9 

95.6 

| 98.8 

98.3 

89.8 

93.8 

Best Trial 
Accuracy Score 

99.5 

98.2 

94.7 

97.1 

| 99.6 

98.9 

98.2 

98.6 

Average 

Efficiency 

Score 

98.3 

83.6 

83.2 

89.2 

| 89.5 

46.5 

68.8 

74.0 

Best Trial 
Efficiency 
Score 

98.6 

97.9 

87.0 

90.1 

| 99.2 

j 

62.3 

82.5 

83.2 


Data based on two subjects. Data for one subject was not recorded by the ITS . 
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Table 2 Continued. Mean Performance for Loaded Part Tasks. 


Variable 

INTEG 
LOC 1 

INTEG 
LOC 2 

INTEG 
LOC 3 

INTEG 
LOC 4 

# of Trials 
Required to Pass 

2.0 

2 . 0 2 

2 . 0 3 

2 . 0 3 

# of Trials 
Failed 

0.0 

0.0 

0.0 

0.0 

Average Time per 
Trial (Sec.) 

130.2 

199.8 

175.0 

155.5 

Best Trial Time 
(Sec. ) 

127.3 

191.0 

159.5 

153.5 

Average 
Total Score 

87.0 

82.0 

89.3 

87.2 

Best Trial 
Total Score 

90.0 

82.6 

91.5 

87.6 

Average 

Accuracy Score 

97.5 

96.8 

97.2 

96.7 

Best Trial 
Accuracy Score 

98.2 

96.9 

97.9 

96.3 

Average 

Efficiency Score 

55.5 

37.9 

65.7 

00 

. 

00 

in 

Best Trial 
Efficiency Score 

65.2 

39.8 

72.7 

61.3 


Data reported for three subjects; however, one subject 
completed only one trial thus did not pass this LOC. 
His data was excluded from the # of Trials Required 
to Pass and # of Trials Failed measures. 

Data provided only for two subjects. One subject 
did not complete LOC 3 and 4 . 


3 
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Table 3. Mean Performance for End Effector Part Tasks. 


Variable 

THC 
LOC 1 

THC 
LOC 2 

THC 
LOC 3 

THC 
LOC 4 

RHC 
LOC 1 

RHC 
LOC 2 

RHC 
LOC 3 

RHC 
LOC 4 

# of Trials 
Required to 
Pass 

7 . 0 1 

4.7 

9.3 

10. 7 2 

! 

— 

— 

2 .0 1 

# of Trials 
Failed 

1.0 

1.7 

2.7 

6.0 

i 

— 

— 

0.0 

Average Time 
per Trial 
(Sec . ) 

22.7 

51.3 

61.9 

103.0 

! 



73.5 

Best Trial Time 
(Sec. ) 

18.0 

34.3 

43.7 

80.5 

! — 

— 

— 

75.0 

Average 
Total Score 

94.3 

80.8 

82.5 

59.6 

I — 

— 

— 

90.2 

Best Trial 
Total Score 

99.3 

93.1 

92.2 

90.4 

i 

— 

— 

90.5 

Average 

Accuracy Score 

97.6 

85.56 

89.5 

68.5 

\ — 

— 

— 

96.3 

Best Trial 
Accuracy Score 

99.5 

94.9 

95.6 

96.9 

| 

— 

— 

95.8 

Average 

Efficiency 

Score 

84.3 

66.4 

61.7 

32.7 

| 

j 

i 



71.7 

Best Trial 
Efficiency 
Score 

98.8 

88.1 

82 .3 

70.9 

1 

" 

" 

74.4 


1 Only one subject completed this LOC, 

2 Only two subjects performed trials in this LOC; neither passed this LOC. 


Data on all performance measures is provided for these two subjects. 
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Table 3 Continued. Mean Performance for End Effector Part Tasks. 


Variable 

INTEG 
LOC 1 

INTEG 
LOC 2 

INTEG 
LOC 3 

INTEG 
LOC 4 

# of Trials 
Required to Pass 

2 . 0 1 

2 . 0 1 

13. 0 1 

— 

# of Trials 
Failed 

0.0 

0.0 

1.7 

— 

Average Time per 
Trial (Sec.) 

59.5 

109.0 

146.4 

— 

Best Trial Time 
(Sec . ) 

50.0 

93.0 

114.0 

— 

Average 
Total Score 

86.0 

84.6 

75.7 

— 

Best Trial 
Total Score 

90.1 

91,0 

86.8 

— 

Average 

Accuracy Score 

92.4 

95.2 

89.4 

— 

Best Trial 
Accuracy Score 

93.1 

95.5 

94.3 

— 

Average 

Efficiency score 

66.2 

52.7 

34.5 

— 

Best Trial 
Efficiency Score 

81.2 

77.4 

64.0 

— 


1 Only one subject completed this LOC. 


Informal Comments and Observations. Subjects provided variety of comments 
and observations relating to the coordinate system part tasks. In the Unloaded 
part tasks, two subjects relied on the ghost arm as the target and were unaware 
of the digitals displayed on the screen until THC LOC 2 in one case and THC LOC 
4 in the other case when the ghost arm no longer appeared. One subject suggested 
we emphasize the digitals displayed to reduce reliance on the ghost arm. 
(However, RMS trainers might not agree with this suggestion, instead recommending 
subjects focus on visualizing the target position and attitude rather than 
relying on the digitals.) At the end of the End Effector part tasks, one subject 
questioned why the ghost arm was removed in LOC 4 for the coordinate systems part 
tasks. He suggested that this focused subjects more on the digitals. However, 
he also reported learning more from the digitals and ignoring the ghost arm. 

Two subjects also had difficulty with the Unloaded RHC part tasks, reporting 
that the RMS often moved in a different direction than expected based on the RHC 
inputs. One subject reported relying more on the digitals due to the difficulty 
in visualizing the target attitude. These subjects had not yet received the 
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briefing on coordinate systems from the shuttle trainer. One subject 
specifically requested instruction from a shuttle trainer on the RHC task. Due 
to scheduling constraints, he did not receive the briefing on coordinate systems, 
including RHC tasks, until comple tion of the Unloaded and Loaded part tasks. 
This subject thought he had learned the wrong technique for RHC tasks, and he 
thought he needed to repeat the previous RHC lessons because he had been working 
with the digitals rather than trying to visualize the target orientation of the 
RMS. This subject also reported that early coaching with a Shuttle trainer could 
help ensure subjects learn the correct approach and further suggested that, it 
might be appropriate to learn the basics on the MDF and other simulators and then 
practice on the ITS. 

One subject received the briefing on coordinate systems prior to beginning 
the Unloaded part task. This subject appeared to experience much less difficulty 
in performing RHC tasks based on his comments versus the other subjects ' 
comments. However, this subject also found the RHC tasks more difficult than the 
THC tasks. Further, he reported that he felt the ITS focused the individual on 
digitals alone. He did not know how realistic this was in relation to RMS tasks 
but felt that the information was valuable at his stage of learning. This 
subject reported that seeing the changes in the digitals and the corresponding 
motion on the screen was useful. 

The experimenter's informal observation was that all three subjects appeared 
to find the RHC tasks more difficult than the THC tasks. However, the subject 
receiving the briefing on coordinate systems in ITS Lesson 1 appeared to 
experience less difficulty with the RHC tasks than the other subjects. The other 
subjects also appeared to find the briefing from the trainer very helpful. The 
experimenter's informal observation was that the Workbook alone was not 
sufficient to prepare subjects for the RHC tasks on the ITS; further, the ITS did 
not provide sufficient information to enable subjects to understand RHC tasks. 

Two subjects commented on the God's eye view displayed in the upper right 
window and questioned its usefulness. One subject commented that it would be 
helpful to know where the cameras were located from which one viewed the RMS. 
He also noted that the "God's eye view" was not an actual camera view available 
on the shuttle. Another subject felt the God's eye view was not appropriate 
because it did not provide a true reference point. These subjects suggested 
having the ITS provide two real views in the upper windows of the display or 
having the ITS provide one view and the subject select the camera view in the 
other window. The two lower windows displayed the control panel and performance 
or ITS use information. (Note: The God's eye view was not provided in LOC 3 and 
LOC 4 tasks but could be accessed using the God's Eye View Hint button.) One 
subject also noticed that if he replaced the God's eye view with another camera 
view, he could not reobtain the God's eye view. 

Two subjects commented on the ability to display the POR in the Loaded part 
tasks. Both subjects had to be told that they could highlight the POR by placing 
the cursor in a window viewing the RMS or shown how to display the wire frame 
version of the RMS. The ITS did not provide this information. However, only one 
subject found highlighting the POR helpful, wanting to see the location of the 
POR on the screen. The other subject reported that highlighting the POR during 
RHC tasks did not provide particularly useful information. (Neither subject had 
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yet received the briefing on coordinate systems.) 

One subject commented that he would like to be able to go back and repeat 
any previous task to obtain additi ona l practice. He did not like the fact that 
the ITS made it mandatory to continue to the next LOC upon passing a task. He 
purposely failed trials to obtain additional practice. 

One subject also reported that the Advanced Organizers, the text information 
and instructions presented prior to each part task, did not provide sufficient 
information to understand or perform the task. The information was sufficient 
to perform the mechanics of the tasks, but subjects appeared to want more 
information to increase their conceptual understanding of the tasks . 

One subject expressed concern about not getting adequate time on the higher 
fidelity simulators (i.e., MDF, SMS, SES) due to the time spent on the ITS. 
(Note: The ITS affected 3 out of 25 RMS 2000 level training modules and replaced 
4 hours out of 49.5 hours usually spent on the MDF, SMS or SES.) 

one subject experienced a computer problem in that the RMS was. moving by 
itself when the ITS program was running. The problem was resolved by exiting and 
reentering the ITS program. Also, one could reset the null position in the hand 
controller by "jiggling” the hand controller. 

One subject reported feeling very pressured in terms of time and accuracy. 
He felt like the machine was going to "beep" at him at any time. This subject 
also expressed a desire to be able to pause the trial to figure something out or 
think about something. 

Two subjects preferred to not complete the End Effector tasks. One subject 
felt he had not mastered the tasks but was still trying to "beat the system", and 
he did not think practice with another coordinate system would change this. The 
second subject felt that the End Effector tasks were unrealistic: they did not 
reflect the use of this mode in actual RMS tasks, and the ITS End Effector tasks 
should be revised. Both subjects were advanced to the procedural part tasks. 

Also, all three subjects at some point reported they were focused more on 
"beating the system" than on learning the tasks. Subjects felt that the ITS 
instructions or tasks cued this attitude. However, one subject also reported 
that the coordinate system lessons were useful in learning how to maneuver the 
hand controllers and in learning the uniqueness of each coordinate system, 
although he felt the LOC 4 tasks were not needed. 

Finally, the experimenter had to explain to each subject how to move the 
parameter dial to get digitals for either position or attitude. The ITS did not 
provide this information. Further, during some part tasks, the ITS processed 
commands s lowly . 

Procedural Tasks : Grapple r Ungrapple, Berth, and Unberth 

Grapple Tasks. Mean performance data for the Grapple procedural part tasks 
are shown in Table 4 . Performance of the Grapple part tasks appeared to be 
relatively easy for subjects, with subjects requiring between 2 and 3 trials on 
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average to advance to the next LOC and failing between 0 and .33 trials on 
average. Subjects generally required between 100 and 180 seconds to perform the 
task, with their best trials requiring between 90 and 160 seconds. Further, they 
performed the task both accurately and efficiently with average scores above 88%. 
safety and camera scores were also high (scores above 95% and 85%, respectively) . 
However, procedure scores were somewhat lower, ranging between 63% and 71% on 
average and between 67% and 87% on their best trials. This might be due to some 
confusion regarding which step in the P2T2 Generic Procedure was the starting 
point in the ITS task as well as hQWL close a view of the grapple fixture was 
required (see informal comments below) . In addition, there were some machine- 
related problems in entering payload ID's which could have resulted in the lower 
procedure scores (see comments below). 

ungrapple Tasks. Mean performance data for the Ungrapple procedural part 
tasks are shown in Table 4 . All three subjects experienced some computer 
problems in entering payload ID f a in the ungrapple task at some point (see 
comments below) . One subject passed LOC 1, experienced computer problems in LOC 
2 and was advanced to the Berth tasks. One subject experienced computer problems 
in LOC 1 and was advanced to the Berth tasks. The third subject experienced 
computer problems in LOC 1 but was able to complete LOC 2 and 3. Subjects were 
able to complete trials but unable to pass trials in which they encountered 
computer problems because of the penalty associated with not entering correct 
payload ID's. unless otherwise noted, the performance data reported includes 
data from any trial a subject was able to complete regardless of computer 
problems . 

Subjects generally found this task easy. Those subjects who did not 
experience computer problems in a given LOC required only two trials to 
successfully pass that LOC. Time required to perform the task ranged from 81 to 
102 seconds. Average total scores ranged from 71% to 98%, although these scores 
included trials in which subjects experienced computer problems. Their best 
total scores were between 83% and 98%. Accuracy scores ranged from 62% to 98%, 
with lower scores probably due to the computer problems. Efficiency was high 
with scores between 89% and 100% in both the average and best trial performance 
data. Safety and camera scores were also high, exceeding 90%. Finally, 
procedure scores were low, again probably due the computer problems. 

Berth Tasks . Mean performance data for the Berth procedural part tasks are 
shown in Table 5. Subjects found this task relatively easy, requiring only 2 
trials to advance to the next LOC and failing no trials. On average, subjects 
required between 3 and 5 minutes to complete a trial. Their total scores 
exceeded 90%. Further, accuracy and efficiency scores were high, all exceeding 
90%. Finally, safety, camera and procedure scores were uniformly high with 
scores exceeding 88% for average and best trial performance. 

Unberth Tasks. Mean performance data for the Unberth procedural part tasks 
are shown in Table 5. Similar to the Berth tasks, subjects found these tasks 
easy, subjects required only two trials to advance to the next LOC and failed 
no trials. Subjects required between 1.5 and 4 minutes to complete a trial for 
average or best trial performance. Total scores exceeded 90%. Similarly, 
accuracy and efficiency scores were high with scores of at least 95%. Subjects' 
safety and procedure scores were also high with minimum scores of 95%. Finally, 
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camera scores were high (between 95% and 100%) in LOC 1 and 2, although they were 
somewhat lower in LOC 3 (60% average; 86% best trial). This probably relates to 
subjects' comments regarding which camera views they prefer and whether such 
views are deemed acceptable by the ITS (see below). 


Table 4 . Mean Performance for Grapple and Ungrapple Part Tasks . 


Variable 

GRAP 
LOC 1 

GRAP 
’ LOC 2 

GRAP 
LOC 3 

UNGR 
LOC 1 

UNGR 
LOC 2 

UNGR 
LOC 3 

# of Trials 
Required to Pass 

3.3 

2.7 

2.0 ] 

2 . 0 1 

2 . 0 2 

2 . 0 3 

# of Trials 
Failed 

0.3 

0.0 

0.0 | 

0.0 

0.0 

0.0 

Average Time per 
Trial (Sec.) 

175.4 

106.0 

132.0 | 

85.1 

81.0 

102.0 

Best Trial Time 
( Sec . ) 

160.7 

106.0 

92.7 

58.3 

89.5 

92.0 

Average 
Total Score 

86.2 

90.6 

85.8 ] 

71.4 

81.6 

98.2 

Best Trial 
Total Score 

91.3 

92.6 

87.7 ] 

! 

83.4 

81.6 

98.3 

Average 

Accuracy score 

88.9 

91.5 

93.7 | 

62.5 

97.5 

87.8 

Best Trial 
Accuracy Score 

91.2 

93.6 

84.8 

98.5 

97.9 

88.4 

Average 

Efficiency Score 

92.8 

98.7 

97.8 | 

89.0 

99.4 

99.8 

Best Trial 
Efficiency Score 

94.5 

98.9 

99.5 

99.8 

99.4 

100.0 

Average 
Safety Score 

92.7 

100.0 

100.0 I 

95.2 

100.0 

100.0 

Best Trial 
Safety Score 

100.0 

100.0 

100.0 1 

100.0 

100.0 

100.0 

Average 

Procedure Score 

70.7 

78.3 

63.3 1 

i 

43.1 

50.0 

100.0 

Best Trial 
Procedure Score 

80.0 

86.7 

66.7 | 

3 

; 

53.3 

i 

j 

50.0 

100.0 

Average 
Camera Score 

93.6 

00 

. 

00 

91.7 i 

94.2 

90.2 

100.0 

Best Trial 
Camera Score 

100.0 

95.6 

100.0 | 

i 

100.0 

i 

90.2 

100.0 
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Note: GRAP indicates Grapple tasks . UNGR indicates Ungrapple tasks. 

Note: A computer problem relating to entering data from the keyboard 

prevented subjects from being able to successfully complete trials in the 
Ungrapple LOC 1 and 2 tasks. The problem was resolved, enabling one subject 
to complete LOC 2 and 3 . 

1 Only one subject passed LOC 1. The other subjects performed 4 and 7 
trials. Their data is excluded from the # of Trials Required to 
Pass and # of Trials Failed measures* 

2 One subject passed LOC 2 One subject completed only one trial; his data 
was excluded from the # Trials to Pass and # Trials Failed measures. 

Only one subject completed LOC 3. 


3 
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Table 5 - Mean Performance for Berth and Unberth Part Tasks . 


Variable 

BERTH 
LOC 1 

BERTH 
LOC 2 

BERTH 
LOC 3 

UNBERTH 
LOC 1 

UNBERTH 
LOC 2 

UNBERTH 
LOC 3 

# of Trials 
Required to Pass 

2.0 

2.0 

2.0 

2.0 

2.0 1 

2 . 0 2 

# of Trials 
Failed 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Average Time per 
Trial (Sec.) 

182.7 

223.8 

282.2 

108.8 

210.0 

240.5 

Best Trial Time 
(Sec . ) 

188.0 

217.7 

291.0 

93.3 

199.0 

225.5 

Average 
Total Score 

93.5 

97.1 

97.5 

98.9 

99.2 

94.5 

Best Trial 
Total Score 

97.1 

97.6 

98.7 

99.6 

99.5 

96.7 

Average 

Accuracy Score 

92.3 

95.8 

96.2 

98.3 

98.8 

98.1 

Best Trial 
Accuracy score 

92.1 

95.9 

97.2 

98.7 

99.6 

98.6 

Average 

Efficiency Score 

94.4 

95.7 

95.6 | 

98.6 

96.1 

95.9 

Best Trial 
Efficiency Score 

94.4 

95.9 

95.3 ] 

99.7 

96.6 

95.9 

Average 
Safety Score 

91.7 

100.0 

95.8 | 

j 

100.0 

100.0 

97.5 

Best Trial 
Safety Score 

100.0 

100.0 

100.0 I 

100.0 

100.0 

95.00 

Average 

Procedure Score 

93.3 

100.0 

100.0 1 

100.0 

100.0 

100.0 

Best Trial 
Procedure score 

100.0 

100.0 

100.0 | 

100.0 

100.0 

100.0 

Average 
Camera Score 

100.0 

88.1 

99.6 j 

1 

i 

95.5 

99.8 

60.20 

Best Trial 
Camera Score 

100.0 

92.5 

ioo.o ! 

1 

100.0 

100.0 

86.00 


* One subject completed only one trial in LOC 2 . His data was excluded 
from the # of Trials Required to Pass and # of Trials Failed measures 
One subject did not complete LOC 3 . .... 



2 
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Informal Comments and Observations. Subjects provided a variety of informal 
comments and observations relating to the Grapple, Ungrapple, Berth, and Unberth 
procedural tasks . 

For the Grapple task, all three subjects noted that the feedback said a 
closer view of the end effector and the grapple fixture was needed even when the 
view was so close that the end effector was out of the picture. Thus, subject 
found the feedback was sometimes confusing and thought the camera view required 
by the ITS was too close. One subject suggested that the ITS include tips on how 
close to move the end effector to the grapple fixture. Finally, one subject 
noted that the ITS implied there is only one right way of setting the cameras and 
that this was not true. 

One subject also reported that the ITS needed to provide more information 
on where the ITS task was starting in relation to the P2T2 Generic Procedures. 
Further, he suggested that a lever should be placed to the right of the master 
alarm on the a8u panel to make the simulator more realistic. 

For all three subjects, the ITS performed very slowly in the Grapple LOC 3 
task. Further, there was some machine-related difficulty in setting the payload 
ID's in the task. 

One subject thought the ITS Grapple task would be particularly useful in 
learning to grapple new, unique payloads, once you have mastered the actual 
grapple procedure. 

A trainer, observing one subject's ITS session, thought the RMS should be 
set up so that one has to line up the RMS directly over the grapple fixture prior 
to beginning the grapple procedure. The trainer also thought the procedural 
tasks provided effective training content to the subjects. Finally, the trainer 
liked the realism provided by the control panel window displayed by the ITS. 

Two subjects had questions regarding the difference between release and 
ungrapple procedures. Further, one subject noted that the release procedure 
checklist wrongly indicates that the RMS is moved to the target position before 
monitoring talkbacks. Also, one subject noted that the P2T2 Generic Procedures 
did not indicate that one should set the ID's to 0 after moving to the post- 
release position. Finally, all three sub jects experienced some computer problems 
in entering payload ID's and were advanced to the next LOC. This occurred in LOC 
1 for two subjects and in LOC 2 for the third subject. 

For the Berthing task, this subject noted that on the Orbiter there is 
another panel that lets the user know where the trunions are in the V-guides and 
further, captures the payload so that you don't have to drive it all the way in. 
This subject also noted that the. ..ITS does not provide depth perception 
information, so one must rely solely on the Y axis. He did not think this 
reflected RMS tasks on the Orbiter. He also felt the ITS training for this task 
should only supplement the regular training. 

Another subject selected the cameras differently than expected by the ITS 
when berthing a payload in the V-guides. This subject selected side cameras so 
that the furthest V-guide could be seen. One could tell the depth by the 
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digitals, but the visuals would also be helpful. This subject preferred Camera 
D although Camera C was the correct choice according to the ITS. However, the 
subject used Camera c because the Camera D lens was too large to focus in on the 
v-guides. This subject also noted that one can split the screen on the Orbiter 
to line up two camera views. The ITS doesn't have this capability. 

One subject completed an MDF class on berthing prior to working on the ITS 
Berthing task. The subject reported that the Berthing task seemed more straight 
forward due to the MDF experience. This subject felt that the MDF lessons on 
Berthing and Unberthing should precede the ITS lessons on these topics. The MDF 
provides more information on v-guides, etc. However, he felt the ITS Grapple and 
Ungrapple tasks could precede the MDF lessons. 

For the Unberthing task, one subject did not like the God's eye view and 
selected an alternate camera. Also, this subject noted that it was interesting 
that the ITS chose cross-bay camera views. He reported that in other RMS 
training, the A and D or B and c ca mera s are used; use of cross-bay camera views 
of the trunions and V-guides is the wrong technique. The experimenter's informal 
observation was that the ITS should indicate to the subject that the camera views 
are adjusted for the subject in LOC 1, and further, the ITS should explain the 
rationale behind its camera view selections. The trainer observing this session 
also noted that this task was less realistic because the dynamic motion is not 
modeled. That is, on the Orbiter one does not have the luxury of using one hand 
to input THC commands and the other to adjust camera views. Rather, both hands 
are needed to input THC and RHC inputs due to the dynamic motion. The trainer 
also reported that it would be helpful to see more of the Orbiter when the 
payload is out of the bay. In actual RMS tasks, one should have camera views of 
the whole RMS. He also suggested using a variety of payloads in the LOC 1 
Unberthing task. 

Procedural Tasks: Recognizing and Visualizing singularities and Reach Limits 

Recognizing Singularities . Mean performance data for the Recognizing 
Singularities procedural part tasks are shown in Table 6. Performance of these 
tasks appeared to be somewhat difficult for subjects. Subjects required between 
2 and 4 trials to advance to the next LOC and failed between 0 and 1 trial on 
average. Subjects required little time to complete a trial (between 4 and 25 
seconds on average). subjects' total scores ranged from 76% to 100% with 
subjects experiencing the most difficulty in LOC 1. This might be due to their 
confusion about whether they were seeing one view of the RMS or three views (see 
comments below) . 

Visualizing Singularities . Mean performance data for the visualizing 
Singularities procedural part tasks are shown in Table 6. Performance of these 
tasks appeared to be slightly more difficult than the previous Recognizing tasks. 
Subjects required between 3 and 6 trials to advance to the next LOC and failed 
between .33 and 2 trials- Subjects required between 2.5 and 5.5 minutes to 
complete a trial. Subjects' total scores were higher in LOC 1 (92%) than in LOC 
2 (86%) and LOC 3 (78.5%), indicating that subjects found the tasks increasingly 
difficult. Efficiency scores remained high throughout the three LOC's, ranging 
between 87% and 90%. Accuracy scores, though, were highest in LOC 1 (92%) and 
declined in LOC 2 (85%) and LOC 3 (78%). 
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Table 6. Mean Performance for Recognizing and Visualizing Singularities Part 
Tasks • 


Variable 

RSING 
LOC 1 

RSING 
LOC 2 

RSING 
LOC 3 

RSING 
LOC 4 

VSING 
LOC 1 

VSING 
LOC 2 

VSING 
LOC 3 

# of Trials 
Required to Pass 

4.0 

2.7 

2.0 

2.0 

2.7 

4.5 1 

6 . 0 1 

# of Trials 
Failed 

1.0 

0.3 

0.0 

0.0 

0.3 

1.5 

2.0 

Average Time per 
Trial (Sec.) 

24.6 

17.7 

3.8 

11.0 

156.7 

183.3 

316.8 

Best Trial Time 
(Sec . ) 

5.7 

6.7 

2.0 

9.0 

83.7 

137.5 

281.0 

Average 
Total Score 

75.7 

88.6 

100.0 

98.7 

91.6 

i 

85.6 

78.5 

Best Trial 
Total Score 

100.0 

99.2 

100.0 

99.4 

| 99.8 

i 

99.8 

91.8 

Average 

Accuracy Score 

83.3 

91.7 

100.0 

100.0 

| 92.1 

84.6 

87.2 

Best Trial 
Accuracy Score 

100.0 

100.0 

100.0 

100.0 

1 100.0 

100.0 

97.5 

Average 

Efficiency Score 

77.7 

79.6 

100.0 

95.0 

1 90.2 

88.6 

86.8 

Best Trial 
Efficiency Score 

100.0 

96.7 

100.0 

97.5 

| 99.0 

99.3 

75.0 


Note: RSING indicates Recognizing Singularities; VSING indicates visualizing 

Singularities . 

1 one subject did not complete Visualizing Singularity LOC 2 or 3. 


Recognizing Reach Limits. Mean performance data for the Recognizing Reach 
Limits procedural part tasks are shown in Table 7 - Due to out-of-date 
specifications and lack of a key piece of information, subjects did not perform 
LOC 6 or LOC 7, involving wrist roll reach limits and all reach limits, 
respectively (see comments below). Performance of these tasks and the 
Recognizing singularities tasks was similar. Subjects required between 2 and 4 
trials to advance to the next LOC and failed between 0 and 1 trial on average. 
Further, they required little time to perform the task (between 14 and 31 seconds 
in LOC 1 through 5). The longer time (74 seconds) shown in LOC 6 reflects the 
difficulty with the wrist roll task mentioned above. 

Subjects total scores generally improved from LOC 1 (73%) to LOC 5 (93%). 
This was more clearly indicated by the low efficiency (56%) and accuracy (78%) 
scores in LOC 1. These scores improved in later LOC's with efficiency scores 
between 71% and 86% and accuracy scores between 82% and 100% in LOC 2 through 5. 
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Visualizing Reach Limits. Mean performance data for the Visualizing Reach 
Limits procedural part tasks are shown in Table 7. Subjects found this task 
somewhat more difficult, requiring between 4 and 7 trials to advance to the next 
LOC and failing between 1 and 3 trials. Further, subjects required between 100 
and 165 seconds to complete a trial. Total scores were somewhat lower than 
observed in other tasks with scores between 76% and 81%. Efficiency scores were 
relatively high, exceeding 87%. Thus, the lower total scores reflected lower 
accuracy in performance. Accuracy scores were between 73% and 79%. 

Table 7 . Mean Performance for Recognizing and Visualizing Reach Limits Part 
Tasks. 


Variable 

RRL 
LOC 1 

RRL 
LOC 2 

RRL 
LOC 3 

RRL 
LOC 4 

RRL 
LOC 5 

RRL 
LOC 6 

# of Trials 
Required to Pass 

3.7 

4.3 

2.3 

2.0 

2.0 

1.5 1 

# of Trials 
Failed 

1.0 

1.0 

0.0 

0.0 

0.7 

1.5 

Average Time per 
Trial ( Sec . ) 

29.0 

16.2 

15.0 

14.2 

31.5 

74.2 

Best Trial Time 
(Sec . ) 

14.0 

11.3 

6.7 

10.0 

15.3 

63.0 

Average 
Total Score 

72.8 

81.7 

95.5 

96.6 

92.7 

3.4 

Best Trial 
Total Score 

96.9 

97.1 

99.4 

98.5 

94.8 

3.4 

Average 

Accuracy Score 

78.3 

82.1 

100.0 

100.0 

100.0 

0.0 

Best Trial 
Accuracy Score 

100.0 

100.0 

100.0 

100.0 

100.0 

0.0 

Average 

Efficiency Score 

56.5 

80.4 

82.1 

86.2 

70.8 

13.7 

Best Trial 
Efficiency Score 

m 

CO 

00 

00 

. 

CO 

97.5 

94.2 

79.2 

13.7 


Note: RRL indicates Recognizing Reach Limits. 


1 One two subjects attempted LOC 6, performing 1 or 2 trials. However, 

the ITS did not provided needed information on joint angles for the wrist 
roll reach limit. Thus, subjects were advanced past both LOC 6 and 7. 
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Table 7 Continued . Mean Performance for Recognizing and Visualizing Reach Limits 
Part Tasks. 


Variable 

VRL 
LOC 1 

VRL 
LOC 2 

VRL 
LOC 3 

# of Trials 
Required to Pass 

4.0 

4.0 

7.5 

# of Trials 
Failed 

1.3 

1.3 

3.0 

Average Time per 
Trial (Sec.) 

100.6 

165.6 

116.5 

Best Trial Time 
(Sec . ) 

58.3 

95.7 

65.0 

Average 
Total Score 

81.2 

78.1 

76.3 

Best Trial 
Total Score 

98.0 

98.3 

99.5 

Average 

78.7 

75.3 

72.9 

Accuracy Score 




Best Trial 
Accuracy Score 

100.0 

98.3 

100.0 

Average 

Efficiency Score 

88.6 

86.7 

86.8 

Best Trial 

92.0 

98.1 

98.0 

Efficiency Score 





NOTE: VRL indicates Visualizing Reach Limits. 


Informal comments and Observations. Subjects provided a variety of informal 
comments and observations relating to the Recognizing and Visualizing 
Singularities and Reach Limits procedural tasks. 

For the Recognizing Singularities task, all three subjects were confused by 
the ITS instructions, thinking that the ITS was displaying three different RMS 
configurations rather than three different views of one RMS configuration. One 
subject also thought that the ITS highlight characteristics of an RMS in a 
singularity to help the user understand why the arm is in a singularity. 

The trainer observing these sessions suggested there was a better way to 
teach individuals about singularities. He suggested that the ITS display the RMS 
in a singularity and demonstrate in which directions it was still able to move 
and in which direction it was no longer able to move. The trainer also noted 
that sometimes movement in more than one direction which could result in a 
singularity although the ITS allowed for only one correct answer. The trainer 
suggested that the instructions should be changed to indicate that the "correct" 
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answer reflected the direction which resulted in the shortest distance to the 
singularity. The trainer also noted one trial which exhibited an RMS 
configuration that had already passed through the singularity; the subject was 
required to move back into the singularity. He suggested that the ITS should 
indicate this. Further, the trainer noted that the ITS deducted 50% for moving 
into a software stop but failed to define a software stop or indicate how to 
avoid it. Finally, the trainer noted that the labeling of which THC or RHC input 
(e.g., +x, -X, +yaw, -pitch) would result in a singularity was confusing. 

One subject had several questions regarding the characteristics (e.g., 
position, attitude, degrees of rotation) of different singularities and reach 
limits . He also did not understand the difference between singularities and 
reach limits and asked about the difference between reach limits, software stops, 
and hardware stops. The trainer observing the session demonstrated singularities 
and reach limits using a mechanical model of the RMS. 

One subject also found it difficult to see the angles displayed by the ITS 
due to the quality of the graphics. This subject also reported that he would 
prefer to receive feedback providing an explanation rather than efficiency and 
accuracy scores. For example, it would be helpful if the ITS stated "the RMS is 

in a reach limit because the angle is degrees and the limits are 

degrees and degrees. He would like an explanation provided by the ITS for 

both correct and incorrect responses. This subject found having the trainer 
available to answer questions and seeing the demonstration on the mechanical 
model very helpful. 

For the Visualizing Singularities or Reach Limits tasks, one subject 
commented that the ITS does not provide information on how well one drove the 
RMS. 


For the Recognizing Reach Limits task, the instructions for the wrist roll 
reach limit were incorrect. The instructions referred to Spec 96; this needs to 
be updated to 0121. Because Spec 96 was out of date and did not provide joint 
angles, the ITS was advanced to the visualization task. Thus, subjects did not 
complete LOC 6 (although two subjects performed 1 or 2 trials) or LOC 7. 

For the Visualizing Reach Limits tasks, there was some question about 
whether the ITS answers were always correct. The subject noted that in one trial 
the ITS gave a negative limit as the correct answer when he observed a positive 
wrist yaw reach limit. Further, the subject noted that in some cases there are 
multiple right answers regarding which THC or RHC input would drive the RMS into 
a reach limit. The ITS, however, only allows for one correct answer. The 
subject also commented that an option to exit the lesson should be provided if 
the subject does not think he is understanding the concepts. Further, the 
trainer observing these sessions noted at least one case where the ITS said a 
positive wrist yaw was negative. There was some question regarding whether a 
wrist pitch trial was also labelled incorrectly by the ITS. 

Survey of Content Knowledge 

Two subjects answered 30 out of 33 questions correctly on the survey of 
content knowledge, with 1 error each in the questions relating to coordinate 
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systems , task procedures , and singularities. The third subject answered 24 out 
of 33 questions correctly. This subject made 3 errors in the questions relating 
to coordinate systems , 4 errors in the procedures questions, and 2 errors 
relating to singularities* The first two subjects completed the survey 
approximately 4 weeks after the final ITS lesson; the third subject completed the 
survey approximately 3 weeks after the final ITS lesson. 

Transfer Performance Measures 


Two subjects were observed on 4 of the 5 tasks: fly-to positions/attitudes, 
grapple, ungrapple, berth, and unberth procedures. Due to time constraints, 
these two subjects did not perform the grapple procedure during the SES session 
in which they were observed. Further, the training content of the SES session 
was revised for the third subject to more adequately address his training needs. 
Thus, data was only available for the grapple procedure for the third subject. 

For the fly-to positions/attitudes task, four raters observed that one 
subject used multiaxis maneuvers sometimes or most of the time while a second 
subject did not use them at all. Both subjects maneuvered in vernier rate within 
10 feet of structures. Both subjects hit either a reach limit or a singularity 
while performing the fly-to. One subject moved the RMS with 12 inches and 10 
degrees of the target position and attitude by using the digitals. The other 
subject did not use the digitals and moved the RMS to a final position/attitude 
more than 12 inches and 10 degrees from the target. Both subjects required 
between 25 and 30 minutes to complete the task and followed either an inefficient 
or acceptable path rather than an efficient path. 

Two raters observed one subject performing the grapple procedure. This 
subject performed the correct sequence of steps and maneuvered in vernier rate 
within 10 feet of structures. He hit no singularities or reach limits. The 
subject moved the RMS to within 12 inches and 10 degrees of the target using an 
inefficient path but good camera views of the payload and orbiter. He required 
approximately 20 minutes to perform the task. 

Subjects performing the grapple, ungrapple, berth, and unberth tasks always 
followed the correct sequence by using the manual provided in the SES. Indeed, 
one purpose of the SES session was to in struct students in the appropriate use 
of the manual and to describe the information available in it. 

For the ungrapple task, a release procedure was observed. Four raters 
observed that both subjects performed the correct sequence of steps. However, 
only one subject maneuvered within vernier rate within 10 feet of structures. 
One subject also hit both a reach limit and a singularity. By using digitals, 
both subjects maneuvered within 12 inches and 10 degrees of the target 
position/attitude. Both subjects were rated as following either an acceptable 
or efficient path and using good camera views. Finally, subjects required 
approximately 5 minutes to complete the task. 

For the berth task, the four raters observed that both subjects performed 
the correct sequence of steps and moved in vernier rate within 10 feet of 
structures. Subjects did not fly into any reach limits or singularities. 
Subjects maneuvered within 12 inches and 10 degrees of the target 
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position/attitude , using an efficient path, and good camera views. One rater 
noted that one subject performed the task particularly well, scanning constantly, 
using multiaxis maneuvers to keep the drift small, and switching frequently from 
translation to rotation digitals* 

For the unberth task, raters observed that both subjects followed the 
correct sequence of steps, moved in vernier rate within 10 feet of structures, 
and flew into no reach limits or singularities* Raters also observed that 
subjects maneuvered within 12 inches and 10 degrees of the target 
position/attitude but indicated that this was too great a tolerance for this 
task. Raters observed that subjects followed an efficient path and used good 
camera views. Subjects required between 20 and 25 minutes to complete the task. 

As a final note, the trainers observing the SES sessions informally 
commented that they did not perceive a performance difference between subjects 
who had completed the ITS -modified training and other astronauts they had trained 
previously. 


Qualitative Data 

Subject Opinion Survey. The subjects generally agreed that the ITS is a 
useful supplement to the training provided on the higher fidelity simulators (M 
= 4.7). More importantly, subjects felt that an ITS (assuming adequate task 
instructions and accurate content) can be an effective training supplement to the 
higher fidelity simulators (M * 5.7). However, subjects did not agree on when 
in the training flow an ITS would be most effective. One subject thought an ITS 
would be most useful after some initial use of the high fidelity simulators . 
Another subject thought an ITS would be most useful as a reviewing tool following 
RMS 2000 level training. The third subject thought an ITS could be useful prior 
to beginning RMS 2000 level training, simultaneously with use of the high 
fidelity simulators, or as a reviewing tool following RMS 2000 level training. 
One subject commented that a good ITS could be built to supplement RMS training 
and teach basic skills. Another subject thought that the current ITS could with 
modest upgrading be used as a reviewing tool following RMS 2000 level training, 
but for initial training an ITS would need to provide much more comprehensive 
analysis of students' performance in comparison to the desired performance. In 
addition, he thought more detailed and explicit hints would be helpful. 

The subjects also thought that the ITS they used was at least moderately 
effective in conveying training content (M = 4*3) and providing basic concepts 
relevant to RMS use (M = 4.0). One subject added that the ITS was better at 
visualizing concepts than explaining how to move the RMS in 3 dimensions . 
Further, subjects felt they had learned the ITS tasks trained rather well (M = 
5.3). One concern was that subjects thought the task instructions relating to 
operating the ITS could be better (M * 3.3). Subjects reported often needing 
assistance from an experimenter or trainer to complete ITS tasks (M - 6.0) and 
often being unable to complete ITS tasks relying only on ITS task instructions, 
the Workbook, and the PDRS P2T2 Generic Procedures (M = 3.3). The additional 
information required often related to characteristics of the ITS itself . 

subjects liked working on the ITS moderately well (M = 3.7), although one 
subject liked working on the ITS much better (5.0) after receiving the briefing 



Training Effectiveness 


31 

on coordinate systems than before (3,0)* Further, one subject found the ITS 
tasks instructive and fun in general, although he found the rotation tasks in 
orbiter Unloaded particularly frustrating and this difficulty is reflected in all 
three subjects' data. Two subjects felt very comfortable working on the ITS in 
general (M = 5.5) and working on it independently for short periods of time (M 
= 5.0), although one subject was not comfortable (2.0). All three subjects found 
it convenient to use the RMS (easy to schedule, etc.) (M = 5.7). Finally, 
subjects reported investing much effort in trying to successfully complete ITS 
tasks (M - 5.7) although they also thought the ITS encouraged trying to beat the 
system rather than understand the concepts (M = 6.3). 

Trainers ' Opinions . Two trainers who possessed Space Shuttle and/or space 
Station training experience provided a variety of informal comments and 
observations relating to the ITS. 

Determining ITS Purpose and Goals* A major issue for the trainers was that 
prior to developing or modifying an ITS, trainers need to decide what the ITS 
should teach. They thought that trainers had not adequately determined the 
purpose of the current ITS and thus , the current ITS might have been too 
ambitious, perhaps teaching content that was too advanced. For future ITS 
development or modification, the trainers thought it was very important to know 
what should be taught in terms of the technical content, assumptions to be made 
about students (i.e., the student model), and how ITS content should be taught 
(i.e., the instructor model). 

Cognitive versus Psychomotor Training. Moreover, the trainers thought the 
system should be training and assessing knowledge rather than performance (which 
includes skills and knowledge) . They did not think the ITS should be a 
psychomotor trainer. Rather, the ITS should do more cognitive training, and 
conceptual knowledge should be evaluated. 

The trainers noted that the kinematic system modelled in the current ITS 
restricted the ITS to low level training. The trainers thought that the ITS 
would need a higher level (higher fidelity) simulation in order to provide higher 
level training. 

Level of Task Difficulty. The trainers thought that the ITS in its current 
form assumed students had some previous RMS experience, perhaps assuming too much 
knowledge on the part of the student s. The current ITS was much more effective 
when an instructor was present to facilitate the training, students found it 
difficult to learn from the ITS alone, needing more substantial instructional 
content than provided by the current system. The trainers suggested that if the 
ITS was designed to provide more advanced training content, it should offer some 
remedial training or more substantive help messages addressing prerequisite 
information and skills. They also thought that the online help messages for the 
ITS tasks needed to be more informative. 

Similarly, the trainers thought the tasks in the ITS were at times 
inappropriate. For example, the ITS asked students to perform a task in a given 
mode whereas in reality one would use the easiest mode available for performing 
any given task. Also, one would use multiple modes if this made performance of 
the task easier. They suggested tha t the ITS use more plausible and simpler 
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tasks in both demonstrations and task practice. 

Finally, trainers would like to see intelligence in the part tasks as well 
as in the whole tasks. Currently, the intelligent component of the ITS resides 
within the whole tasks. However, they felt the tutoring provided even in the 
whole tasks was insufficient. 


Determining ITS Instructional Strategies. The trainers thought that more 
effort should be devoted to determining how best to teach the ITS content. The 
trainers thought that we needed better "education” in the ITS. The software is 
very sophisticated and the educational philosophy and techniques used are too 
simple in comparison. The trainers had several suggestions regarding 
instructional strategy. These suggestions were based on using multiple modes of 
instruction. For example, a three step approach could be used: provide 
substantial initial information regarding a task, provide a demonstration of the 
task, and then allow students an opportunity to perform the task. A more 
comprehensive approach might be to have the student read about the task, see it, 
hear /read a lecture about it, be tested on it, review it, then finally perform 
it. Also, in the future one could use more modelling: see it done, visualize 
it, then do it. 

Student Modelling: Evaluation and Remediation. The trainers thought the 
ITS needed to more effectively evaluate and remediate students right from the 
start. That is, the ITS should examine what training (ITS or other) has been 
completed by the student, how adequately the student performs the task trained, 
and then predict what should be taught next on the basis of this information. 
Further, the system should incorporate more repetition of information that needs 
to be retained. In addition, the trainers recommended that the student model be 
individualized — it should not assume that all students are the same. The ITS 
should assume that students bring different skills to the task and learn in 
different ways. 

Coaching and Feedback. A related issue was the coaching and feedback 
function of the ITS. That is, the trainers thought that an ITS needs a stronger 
coaching function in order to be a true ITS. They felt that the coaching 
provided (i.e., help messages, hints) was insufficient in both the part and whole 
tasks. They suggested that to devise a better coaching function, one could 
collect verbal protocols from subjects to determine the kinds of questions asked. 
The trainers also thought that it would be helpful in designing and/or modifying 
an ITS to spend more time with trainers to develop the training content and 
coaching information. The trainers noted that they did not have much time to 
give but that they needed to give more time in the future during the development 
or modification of an ITS to improve the coaching function. 

The trainers were also concern ed about the feedback provided. They thought 
the performance numbers were not helpful. More specifically, the system would 
be more effective if it provided more detailed information, especially on 
performance inadequacies. They suggested that the system could provide 
qualitative descriptions of performance, such as a "good” /"poor" evaluation 
followed by narrative information on what was done wrong and what to do the next 
time . They also suggested that a pictorial evaluation such as bar graphs would 
be helpful, perhaps comparing the student with other students or the best ones. 
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Also, the trainers thought students should receive positive feedback when 
they perform a task correctly to reinforce, motivate, and inform the student. 
However, the feedback in the current system was more negative in tone. Moreover, 
trainers thought the system needed to provide more specific positive and negative 
feedback. The trainers were also concerned that the ITS puts students into an 
endless loop. That is, the ITS always told students they needed more training, 
i.e., advanced them to the next LOC. They thought the ITS should provide clearer 
endings to parts of training. (This sounds like an issue relating to student 
motivation.) Further, the trainers thought the ITS required too much time, 
noting that if students passed a few trials then failed one trial, they had to 
begin again. 

Other Issues. The trainers suggested that students needed to learn camera 
viewing skills earlier. For example, camera viewing skills would have 
facilitated subject's performance "of “coordinate systems tasks. A broader issue 
was that the ITS was able to do many things that subjects did not know about. 
For example, changing camera views could have facilitated coordinate system 
tasks, but subjects were not told they could manipulate camera views or how to 
use the cameras prior to or during these tasks. 

The trainers also commented that the God's Eye View should be used only in 
demonstrations and in help messages. It should not be used in tasks performed 
by students. Further, trainers noted that the ITS should be able to handle 
multiple correct ways of performing the task. Finally, at higher levels of ITS 
tasks, the trainers thought the ITS should allow for more "free playing". This 
type of activity was only allowed in simulation mode in the current system. 

Discussion and Conclusions 

The results indicated that subjects were generally able to learn RMS tasks 
taught by the ITS. Two of the three subjects were able to correctly answer 33 
out of 36 conceptual question relating to RMS use, indicating that subjects 
gained substantial conceptual knowledge using the ITS. Further, ITS performance 
data indicated that subjects successfully completed ITS lessons on coordinate 
systems, procedural tasks such as grapple, ungrapple, berth, and unberth, and 
singularities and reach limits. Similarly, transfer task performance data 
indicated that subjects were able to perform the 5 tasks assessed (fly-to's, 
grapple, ungrapple, berth, and unberth), and in addition, the trainers present 
during SES sessions perceived that subjects receiving iTS-modified training 
performed as well as other astronauts they had trained. Thus, the ITS-modified 
training was not perceived to be dysfunctional. Indeed, the knowledge gained 
using the ITS helped subjects perforin tasks on the SES, providing evidence of 
positive transfer of training between the ITS and the SES. 

Further, subjects reported that the ITS presented training content and basic 
RMS concepts moderately effectively. Subjects thought they had learned to 
perform the ITS tasks rather well although there was some concern that the ITS 
encouraged them to try to beat the system rather than learn the concepts. In 
addition, subjects viewed the current ITS as at least moderately effective in 
supplementing training and perceived a well-designed ITS to be a very useful and 
effective supplement to training on higher fidelity simulators. Finally, 
trainers reported that a well-designed ITS could be a very effective supplement 
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to training on higher fidelity simulators, although they had several suggestions 
for improving the current ITS. 

Some caution must be used in interpreting these results in view of the small 
sample, i.e., two trainers and three astronauts. Specifically, it is unclear to 
what extent the performance data, opinions, and suggestions obtained generalize 
to other trainers and astronauts. For example, other trainers might not agree 
with all of the opinions stated. Moreover, the astronauts who participated in 
this project were new to the space program. Thus, more experienced astronauts 
might have provided different suggestions for modifying the RMS ITS. 

However, the results were based on a variety of measures, including ITS task 
performance, SES performance, a conceptual knowledge test, and self-reports from 
subjects and trainers. Further, there was substantial agreement across the 
astronauts and trainers participating in the project in terms of performance 
data, opinions, and suggestions. Thus, the convergence of results provides 
evidence in support of the efficacy of ITS's in general, and at least moderate 
support for the current ITS. 

The results also raise several issues. Most importantly, use of the current 
ITS highlighted how critical it is to involve trainers/educators in the design 
of ITS's. Trainers /educators are needed to determine the goals and purpose of 
the ITS, devise the instructional strategy, and determine effective methods to 
evaluate students, provide feedback, and tutor or coach students. While the 
software for ITS's are very sophisticated, more work is needed to ensure that the 
educational/instructional component is of the highest quality. That is, while 
the timing and placement of hints, help, task instructions, and feedback is 
important, one must also ensure that the content of these forms of information 
is accurate and of sufficient detail to most facilitate learning. A strong 
partnership is required between software developers and trainers or task experts 
to ensure that the content of ITS training matches as closely as possible the 
content provided by a good trainer. 

Second, trainers and subjects had a variety of suggestions for improving the 
current ITS or developing new ITS's. A major concern related to the 
instructional strategy used, with suggestions to revise the ITS to incorporate 
a multiple mode strategy, e.g., provide substantial instructions, demonstrate, 
then have subjects perform the task. Trainers and subjects also suggested 
revisions to the feedback and coaching/tutoring provided, indicating a need for 
more descriptive, detailed information and less of a focus on scores. Finally, 
trainers recommended that ITS training should be individualized with the sequence 
of training determined by each individual's needs and abilities. 

Third, an issue raised by the iTS-modif ied training related to the 
sensitivity of the performance measures. Subjects were required to perform at 
certain levels to pass a given trial, so subjects' scores were generally high. 
High scores imply that subjects have mastered a given task. However, subjects 
reported differential amounts of difficulty with various tasks and differing 
levels of perceived competence. This might be due to individual's perceptions 
of themselves. Alternately, it might reflect a lack of sensitivity in the 
performance measures to adequately differentiate among individuals. For example, 
the performance criteria could be very lenient or very stringent. Thus, whether 
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passing a trial indicates that one has mastered the task or demonstrated some 
minimum level of competence depends on the stringency of the performance 
criteria. The ability to adjust the performance criteria makes the system more 
flexible, but it might make it more difficult to determine whether one has 
mastered the trial or merely demonstrated minimal competency. Some additional 
consideration might be needed to determine what levels of performance constitute 
novice, competent, and expert performance. In addition, given the subjects' and 
trainers' comments, consideration should be given to whether other additional 
performance information might be helpful, such as more descriptive feedback. 

In conclusion, the RMS ITS appeared to be moderately effective in conveying 
the content it was designed to present, and individuals were able to learn that 
content. Thus, evidence was provided for the efficacy of this ITS. The results 
also highlight two very important points. First, the use of diverse measures, 
such as performance data, conceptual tests, performance transfer tests, and self- 
reports, can provide strong evidence in evaluating training systems such as 
ITS's, much stronger and more fine-grained evidence than could be obtained using 
one or a few measures. Indeed, the convergence of results in the current study 
enabled stronger conclusions relating to the ITS. Second, the number and variety 
of suggestions for improvements to the ITS indicated in the strongest possible 
terms the need for involving trainers and task experts in the development of 
ITS's. A strong partnership is needed between software developers and trainers 
or educators. The development of software for ITS's is very sophisticated. 
However, the philosophies, methods, and techniques underlying education and 
training are equally sophisticated. Thus, software developers have the potential 
to provide more effective training tools by drawing on the expertise of trainers 
or educators. The costs in terms of time required to involve trainers is far 
overshadowed by the potential benefits. 
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[ Note: This PDRS Operations Checklist is specifically designed for 
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will use for specific missions. Therefore, information given in 
this training supplement that does not normally appear in official 
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GENERIC 

A7U 


GRAPPLE 

1. SETUP 

CCTV - Config for grapple 
- RMS Wrist, zoom out 

SM 94 PDRS CONTROL 


/PL ID - ITEM 3 - 0 EXEC 

✓ INIT ID - ITEM 24 - 0 EXEC 

2. MNVR TO PRE-GRAPPLE POSITION 

RATE - as reqd (VERN within 10 ft) 

BRAKES - OFF (tb-OFF) 

MODE - ORB UNL, ENTER 

Mnvr to Pre-grapple position 

[Pre-grapple position is defined as having the 
EE approximately 5 feet from the grapple 
fixture, lined up with target. In this case, 
pre-grapple has the following coordinates: 


X 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 

-908 

13.7 

-4%. 7 

270 

0 

180.5 

0 

SY 

SP 

EP 

WP 

WY 

WR 







. 


BRAKES - ON (tb-ON) 
MODE - not DIRECT 
JOINT - CRIT TEMP 


FS 1-1 PDRS/P2T2 Intelligent Trainer 


i.aod -sn 




A7U 
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/CCTV - Config for grapple 

On MCC Go for grapple, 

DAP: VRCS or free drift 

[assume you have rec’d] 

RATE - VERN (RATE MIN tb-ON) 

BRAKES - OFF (tb-OFF) 

MOOE - END EFF, ENTER 

Mnvr to grapple envelope 


_ CAUTION 

Monitor EE tb timing to 
prevent EE motor burnout 


EE MODE - AUTO 

EE CAPTURE sw - depress (mom) 

RIGID CLOSE CAPTURE rn rTTrni ttucc 


□ □ □ 

i i i tnu i iriLJ \ uuuai/. 

CAPTURE tb - gray, then 

DERIQO OPEN 

EXTEND 

CLOSE tb - gray, 3 sec max. 

a b 

[ 0 

then 

RIGID tb - gray, 25 sec max 

EE MODE 

- OFF 


BRAKES 

- ON (tb-ON) 

DAP: as reqd 



SM 94 PDRS CONTROL 


PL ID - ITEM 3 +1 EXEC 

IN IT ID - ITEM 24 +1 EXEC 

Record POS/ATT and JOINT ANGLES 


X 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 








SY 

SP 

EP 

WP 

WY 

WR 









FS 1-2 PDRS/P2T2 Intelligent Trainer 
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[GENERIC] UNBERTH 

[Note: "Low Hover" is a specific position where the payload is directly 

over the V-guides and Z =* -650. Therefore, this value is both generic 
and payload-specific and is defined below for each payload you may have 
to unberth during P2T2 training.] 

1. SETUP 

Review LOADED (Cue Card, P2T2-LOAOED/ 

RELEASE ] 

2. MNVR TO LOW HOVER POSITION 

A7U CCTV - Config for Mnvr to Low Hover 

/RATE - VERN (RATE MIN tb-on) 

BRAKES - OFF (tb-OFF) 

MODE - ORB LD, ENTER 

Mnvr payload to Z - -650 (LOW HOVER) 

SPAS I: 


X 

u 

z 



ROLL 

PL ID 


m 




360.0 

1 

SY 

SP 

EP 

UP 

WY 

WR 


-42.1 

83.7 

-88.1 

-98.9 

14.3 

-116.3 


GRO: 


X 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 

-1089.5 

0.6 

-650.0 

0.3 

339.5 

0.0 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-18.5 

74.8 

-88.4 

-93.0 

-10.0 

-51.8 



HST: 



n 




ROLL 

PL ID 

2223 


-650.0 



0.0 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-37.4 

92.2 

-85.2 



ESS 



FS 1-3 PDRS/P2T2 Intelligent Trainer 
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LDEF: 
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X 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 

-892.1 

-14.2 

-650.0 

0.4 

359.7 

90.0 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-19.2 

97.5 

-93.3 

-78.5 

-37.8 

-135.9 


SPARTANH: 






X 

Y 

z - 

PITCH 

YAW 

ROLL 

PL ID 

-712.1 

-0.6 

-550.0 

359.7 

359.6 

360.0 

1 

SY 

SP 

EP 

! WP 

WY 

WR 


-81.3 

93.0 

-119.3 

-112.8 

6.6 

193.8 


IBSS: 







X 

mm 

Z 

PITCH 

[ YAW 

[ ROLL 

PL ID 

-890.4 

-94.7 

-650.0 

0.0 

270.0 

0.0 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-21.8 

86.1 

-82.7 

-95.0 

4.2 

41.3 



BRAKES - ON (tfa-on) 


3. HNVR TO RELEASE POSITION [SPAS] 

A7U CCTV - Config for. Mnvr to Rel position 

DAP: free drift 

RATE - as reqd (VERN within 10 ft) 
BRAKES - OFF (tb-OFF) 

ORB ID to Release position: 


X 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 

-820 

0 

-850 

90 

0 

90 

1 

SY , 

SP 

EP 

WP 

WY 

WR 


-34.7 

+119.4 

-118. 6 

+5.6 

-0.3 

-96.9 



BRAKES - ON (tb-ON) 
MODE - not DIRECT 
JOINT - CRIT TEMP 


FS 1-4 


P0RS/P2T2 Intelligent Trainer 


i.aoo - sn 




















SPAS 


A7U 
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RELEASE 

1. SETUP 


Review RELEASE (Cue Card, P2T2-L0A0ED/ 
RELEASE ) 

/POS/ATT and JOINT ANGLES [SPAS] 


X 

Y 

z 

PITCH 

YAW 

ROLL 

PL ID 

-820 

0- 

-850 

90 

0 

90 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-34.7 

+119.4 

-118.6 

+5.6 

-0.3 

-96.9 


/SAFING tb - gray 
PARAM sel - JOINT ANGLE 

CCTV - RMS Wrist, zoom out 

2. RELEASE 

On MCC Go for Release 

/RATE - VERN (RATE MIN tb-on) 

/BRAKES - OFF (tb-OFF), unless DIRECT/BACKUP 

MODE - END EFF, ENTER 

DAP: free drift 

EE MODE - AUTO 

EE RELEASE sw - depress (mom) 

[ When OPEN tb - gray: 


Mnvr arm clear of GF, payload to: 


X 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 

-806.1 

-81.0 

-862.3 

268.5 

270.0 

89.0 



Note: It is more important to pull straight back on 
the arm to about 5 ft. in relation to the GF than to 
achieve these coordinates. ] 


FS 1-5 PDRS/P2T2 Intelligent Trainer 


i.aoo -sn 
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RIGID CLOSE 



DER1GI0 OPEN 

□ □ 


BRAKES - 
EE MODE 
MODE 

JOINT - 


FS 1-6 


CAPTURE 



EXTEND 

□ 


CRITICAL TIMES (28 sec total) : 
OERIGID tb - gray, 5 sec max, 
then 

OPEN tb - gray, 3 sec max, 
then 

EXTEND tb - gray, 20 sec max 


ON (tb-ON) 
OFF 

not DIRECT 
CRIT TEMP 


PDRS/P2T2 Intelligent Trainer 



A7U 
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[ 3. MNVR TO PRE-CRADLE. POSITION 

CCTV - Config for Mnvr to Pre-Cradle 

SM 94 PORS CONTROL 


50 


PL ID - ITEM 3 +0 EXEC 

IN IT ID - ITEM 24 +0 EXEC 

/RATE - as reqd (VERN within 10 ft) 
✓ BRAKES - OFF (tb-OFF) 

MODE - ORB UNL, ENTER 


SPAS I: 


X 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 

-1261.2 

-145.7 

-551.4 

4.7 

1.7 

359.9 

0 

SY 

SP 

EP 

WP 

WY 

WR 


0.0 

25.0 

-25.0 

5.0 

0.0 

0.0 


BRAKES - ON (tb-ON) 
MODE - not DIRECT 
JOINT - CRIT TEMP 


FS 1-7 


pnR9/P?T2 Intel liqent Trainer 


J.aoo 'S'f 



SPAS POISE FOR CAPTURE 
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1. SETUP 

A7U Config CCTVs as reqd 

SH 94 PDRS CONTROL 

/ PL ID - ITEM 3 +0 EXEC 

✓ IN IT ID - ITEM 24 +0 EXEC 

2. POISE FOR CAPTURE 
RATE - as reqd 

ORB UNL to poise for capture (/ ITEMs 18-25) 


x 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 

-860. C 

-35.0 

-768.0 

44.0 

291.0 

226.5 

0 

SY 

SP 

EP 

WP 

WY 

WR 


-36.5 

111.2 

-102.0 

-14.7 

-38.3 

195.4 


BRAKES - ON (tb-ON) 
/MODE - not DIRECT 
JOINT - CRIT TEMP 


FS 1-8 PDRS/P2T2 Intelligent Trainer 


i.aod 'S'n 
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SPAS CAPTURE 

1. SETUP 

/Target overlays attached as needed 

SM 94 PDRS CONTROL 

/PL ID - ITEM 3 - 0 

/IN IT ID - .ITEM 24 - 0 

Review RHS-CAPTURE/LOADED (Cue Card) 

2. CAPTURE 

CAUTION 

Monitor EE tb timing to 
prevent EE motor burnout 


A7U 


CCTV, RMS/Wrist - zoom out 

RATE - VERN (RATE MIN tb-ON) 

BRAKES - OFF (tb-OFF) 

MODE - END EFF, ENTER 

When grapple fixture in view and stable, 
DAP: free drift 

EE MODE - AUTO 
Mnvr to GF 

EE CAPTURE sw - depress (mom) 


□ RIGID CLOSE CAPTURE rR[TTCAL TIMES' 128 sec total) : 

| I I CAPTURE tb - gray, then 

CLOSE tb - gray, 3 sec max. 


OERtGlD OPEN EXTEND 


then 

RIGID tb - gray, 25 sec max 


1-q 


DnDC/D'JT? Intel! inont Trainer 


j.aoo s'n 
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EE MODE - OFF 

BRAKES - ON (tb-ON) 

/ MODE - not DIRECT 

JOINT - CRIT TEMP 


SM 94 PDRS CONTROL 


PL ID - ITEM 3 +1 EXEC 

IN IT ID - ITEM 24 +1 EXEC 


FS 1-10 


P0RS/P2T2 Intelligent Trainer 


i.aoo -sn 
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[GENERIC]' BERTH 

1. SETUP 

DAP: 8/AUTO/VERN 

SM 94 PDRS CONTROL 

/PL ID - ITEM 3 +1 

/ IN IT ID - ITEM 24 +1 

CCTV - Point CCTV as reqd toward EE/GF interface 

for view of uncmd derig/rel 

2. MNVR TO LOW HOVER 
RATE - as reqd 

ORB LD to Low Hover ( / ITEMs 18-25) : 

SPAS I: 



PL 10 



FS 1-11 PORS/P2T2 Intelligent Trainer 


j.aoo s n 
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LDEF: 


X 

Y 



YAW 

roll 

PL 10 

-892.1 

-14.2 

gj|| 

0.4 

359.7 

90.0 

1 

SY 

SP 

EP 

VIP 

WY 

WR 


-19.2 

97.5 

-93.3 

-78.5 

sm 




SPARTANH: 


X 

Y 

Z 

PITCH 

YAW 

ROLL 

PL ID 

-712.1 

-0.6 

-550.0 

359.7 

359.6 

360.0 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-81.3 

93.0 

-119.3 

-112 . i 

6.6 

193.8 


I8SS: 


nr 

Y 

Z 

PITCH 

YAW 

ROLL 

PL 10 

ME 

EM 


0.0 

SK 

KKI 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-21 . 8 

86.1 

-82.7 

-95.0 

4.2 

41.3 


BRAKES - ON (tb-ON) 

BERTH 

CCTV - Config for berth 

RATE - as reqd (VERN when near structure, -10 ft; 
below Z = -600) 

BRAKES - OFF (tb-OFF) 

MODE ' - ORB LD, ENTER 

DAP: free drift 

Mnvr to berthed position: 

SPAS I: 


X 

fl 

z 

PITCH 

YAW 

■ROLL 

PL ID 

-895.0 

0.0 




360.0 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-29.1 

69.0 



16.9 

-129.5 



FS 1-12 


PflR's/PPT? Intel 1 iapnt Trainor 
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GRO: 



IBSS: 



LDEF: 



SPARTANH: 


X 

mm 

* ] 

PITCH 

YAW 

ROLL 

PL ID 

-712.1 

-0.5 

gggggjl 

359.6 

360.0 

1 

SY 

SP 

EP 

WP 

WY 

WR 


-79.4 

79.7 

-137.8 

-80.9 

8.1 

192.5 


BRAKES - ON (tb-ON) 


CC 1 1 *3 


nf>nc/nOT'? ToFoll innnf fvn nicr 


j.aoo *s‘n 








































4. 


PREP FOR UNGRAPPLE 
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To relieve strain: 

DAP: VRCS or free drift 

BRAKES - OFF (tb-OFF) 

MODE - TEST, ENTER 
Wait 5 sec, then 
BRAKES - ON (tb-ON) 

5. UNGRAPPLE 

A7U CCTV, RMS/Wrist - zoom out 

CRT PL ID - ITEM 3 +0 EXEC 

IN IT ID - ITEM 24 +0 EXEC 

DAP: VRCS or free drift 

A8U RATE - VERN (RATE MIN tb-ON), when within 

10 ft of structure 
BRAKES - OFF (tb-OFF) 

MODE - END EFF, ENTER 


CAUTION 

Monitor EE tb timing to 
prevent EE motor burnout 


If single joint. 

Perform Manual EE Release 

EE MODE - AUTO 

RELEASE sw - depress (mom) 


When OPEN tb - gray, 

Mnvr arm clear of GF, orbiter, payload 


X 

Y 

z 

PITCH 

YAW 

ROLL 

PL ID 

-908.: 

: 13.7 

-494.7 

270.0 

0.0 

180.5 

0 


rigid CLOSE capture CRITICAL TIMES (28 sec total) : 


ill 

OERIGID tb 

- gray, 5 sec max, 
then 

DERIGIO OPEN EXTEND 

OPEN tb 

□ □ □ 

- gray, 3 sec max, 
then 


EXTEND tb 

- gray, 20 sec max 


BRAKES - ON (tb-ON) 


FS 1-14 PDRS/P2T2 Intelligent Trainer 


j.aod s n 
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EE MODE - OFF 

/MODE - not DIRECT 

JOINT - CRIT TEMP 

[ 6. MNVR TO PRE-CRADLE POSITION 

CCTV - Config for Mnvr to Pre-Cradle 

SM 94 PDRS CONTROL 


PL ID - ITEM 3 +0 EXEC 

IN IT ID - ITEM 24 +0 EXEC 


✓ RATE - as reqd (VERN within 10 ft) 

✓ BRAKES - OFF (tb-OFF) 

MODE - ORB UNL, ENTER 



mm 

z 

PITCH 

YAW 

ROLL 

PL ID 


-145.7 



IB 

359.9 

0 

SY 

SP 

EP 

WP 

WY 

WR 









BRAKES - ON (tb-ON) 
MODE - not DIRECT 
JOINT - CRIT TEMP 


FS 1-15 PDRS/P2T2 Intelligent Trainer 


i.aoo - sn 
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Survey of Concepts Regarding RMS Use 

1. In the Body Axis Coordinate System (BACS), if one translated the RMS in 
the -y direction, the RMS would move 

a. toward the nose 

b. starboard 

c. down 

d. port 

2. In the BACS, if one translated the RMS in the -x direction, the RMS 
would move 

a. port 

b . down 

c. toward the nose 

d. toward the tail 

3. In the Rotational Axis Coordinate System (RACS), if one rotated in the 
-yaw direction, 

a. the RMS would rotate to starboard 

b. the RMS would rotate down 

c. the RMS would rotate to port 

d. the RMS would rotate up 

4. In Unloaded mode, the Point of Resolution (POR) is defined to be 

a. the nose of the orblter 

b. the tip of the EE 

c. the origin of the BACS 

d. a point in space, typically a point within a payload 

5. In EE mode, the POR is defined to be 

a. the nose of the orbiter 

b. the tip of the EE 

c. the origin of the BACS 

d. a point in space, typically a point within a payload 

6. In Loaded mode, the POR is defined to be 

a. the nose of the orbiter 

b. the tip of the EE 

c. the origin of the BACS 

d. a point in space, typically a point within a payload 

7. In Payload mode, the POR is defined to be 

a. the nose of the orbiter 

b. the tip of the EE 

C. the origin of the BACS 

d. a point in space, typically a point within a payload 
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For the questions I 8 through 1 23, assume the RMS is centered over the 
payload bay. 

8. In Unloaded mode, if one wanted to move the RMS toward the tail of the 
orbiter, one would move the Translational Hand Controller (THC) 

a . in 

b. up 

c. down 

d . out 

9. In Unloaded mode, if one wanted to move the RMS in the +z direction, one 
would move the THC 

a. down 

b. out 

c. up 

d . in 

10. In Unloaded mode, if one wanted to rotate the RMS toward the starboard 
wing, one would 

a. twist the Rotational Hand C ontro ller (RHC) to the left 

b. twist the RHC to the right 

c. push the RHC forward 

d. pull the RHC back 

11. In Unloaded mode, if one wanted to input a negative roll in the RMS, one 

would 


a. twist the RHC to the left 

b. twist the RHC to the right 

c. push the RHC to the left 

d. pull the RHC to the right 

12. In Loaded mode, if one wanted to move the RMS over the starboard wing, 
one would move the THC 

a . in 

b. right 

c. left 

d . up 

13. In Loaded mode, if one wanted to move the RMS in the +x direction, one 
would move the THC 

a . out 

b. left 

c. right 

d . in 
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14. In Loaded mode, if one wanted to rotate the RMS up, one would 

a. twist the RHC to the left 

b. twist the RHC to the right 

c. push the RHC forward 

d. pull the RHC back 

15. In Loaded mode, if one wanted to input a positive yaw in the RMS, one 
would 

a. twist the RHC to the left 

b. twist the RHC to the right 

c. push the RHC to the left 

d. pull the RHC to the right 

16. The EE is pointing to the port wing with the wrist camera facing the 
payload bay. In EE mode, if one wanted to move the RMS toward the tail 
of the orbiter, one would move the THC 

a. left 

b. in 

c. right 

d . out 

17. The EE is pointing to the tail of the orbiter with the wrist camera 
pointing to the starboard wing. In EE mode, if one wanted to move the 
RMS in the +x direction, one would move the THC 

a. left 

b. in 

c. right 

d. out 

18. The EE is pointing to the tail of the orbiter with the wrist camera 
pointing to the starboard wing. In EE mode, if one wanted to input a 
positive pitch in the RMS, one would 

a. twist the RHC to the left 

b. twist the RHC to the right 

c. push the RHC forward 

d. pull the RHC back 

19. In payload mode, the payload axes are oriented such that +x points down, 
+y points to the starboard wing, and +z points to the tail of the 
orbiter. In relation to the payload axes, if one wanted to move the 
payload in the -x direction, one would move the THC 

a . down 

b. out 

c. up 

d . in 
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20. In payload mode, the payload axes are oriented such that +x points to 

the starboard wing, ty to the tail of the orbiter, and tz points down. 

In payload mode, if one wanted to move the payload in the +x direction, 

one would move the THC 

a. left 

b . in 

c. right 

d . out 

21. In payload mode, the payload axes are oriented such that +x points down, 
+y points to the starboard wing, and tz points to the tail of the 
orbiter. In payload mode, if one wanted to rotate the payload to 
starboard, one would 

a. twist the RHC to the left 

b. twist the RHC to the right 

c. push the RHC forward 

d. pull the RHC back 

22. In payload mode, the payload axes are oriented such that +x points to 

the starboard wing, ty to the tail of the orbiter, and tz points down. 

In payload mode, if one wanted to input a positive roll in the RHS, one 

would 


a. push the RHC to the left 

b. push the RHC to the right 

c. push the RHC forward 

d. pull the RHC back 

23. Pre-grapple position is defined as having the EE about from the 

grapple fixture and lined up with the target. 


24. 


25 . 


a. 10 feet 

b. 2 feet 

c. 5 feet 

d. 120 inches 

When maneuvering to the grapple envelope what mode is the RMS in? 

a. unloaded 

b. EE 

c. loaded 

d. payload 

When maneuvering to low hover position during unberthing a payload, what 
mode is the RMS in? 

a. unloaded 

b. EE 

c. loaded 

d. payload 
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26 . 


One should one switch the rate to vernier when the EE is 
the payload. 

a. 5 feet 

b. 6 feet 

c. 100 inches 

d. 120 inches 


from 


27. Generally speaking and depending on payload dimensions, low hover is a 
specific position where the payload is 

a. directly over the V-guides and z = -650 

b. directly over the V-guides and z = -500 

c. 10 feet above the V-guides and lined up with the target 

d. 5 feet above the V-guides and lined up with the target 

28. When using all the cameras available to view the payload to be grappled, 
the cameras should 

a. view the grapple fixture using the wrist camera 

b. view the grapple fixture from the side using one camera and view 

the grapple fixture using the wrist camera 

c. view the grapple fixture from the side using one camera and view 
the entire payload in relation to the shuttle using another camera 

d. view the grapple fixture using the wrist camera and view the entire 



payload 

in relation to the shuttle using another 

camera 

29. 

Which arm is 

in shoulder yaw singularity? 

(refer to 

figures) 


a. Arm |1 

b. Arm 1 2 




30. 

Which arm is 

in planer pitch singularity? 




a. Arm >2 

b. Arm #3 




31. 

Which arm is 

in wrist yaw singularity? 




a. Arm tl 

b. Arm 13 




32. 

Is Arm 11 in 

a shoulder pitch reach limit? 

Yes 

No 

33. 

Is Arm 12 in 

a wrist pitch reach limit? 

Yes 

No 
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Date : 


Use of multiaxis maneuvers 


1 Not at all 

2 Sometimes 

3 Most of the time or always 


Safety 

Vernier w/in 10 ft 


1 yes 

2 no 


Reach limits 


# flown to 


Singularities 


I flown to 


Accuracy 


1 within 12 inches 

2 more than 12 inches 


1 within 10 degrees 

2 more than 10 degrees 


Efficiency 

Time required 


minutes 

1 inefficient path 

2 acceptable path 

3 efficient path 
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Grapple 

procedures 


Correct sequence of steps 


1 yes 

2 no 

If no, describe out of step 
actions : 


(refer to PDRS Ops Checklist) 


Safety 

Vernier w/in 10 ft 

Reach limits 
Singularities 


1 yes 

2 no 


f flown to 
I flown to 


Accuracy 


1 within 12 inches 

2 more than 12 inches 

1 within 10 degrees 

2 more than 10 degrees 


Efficiency 

Time required 
Path 


minutes 

1 inefficient path 

2 acceptable path 

3 efficient path 



1 

poor views 

of payload/ 


orbiter 


2 

acceptable 

views 

3 

good views 
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Performance Information 


Subject: Date: 


Task 

Ungrapple 

Correct sequence of steps 

i 

yes 

procedures 


2 

no 



If 

no, describe out of step 



actions : 



(refer to PDRS Ops Checklist) 


Safety 




Vernier w/in 10 ft 

1 

yes 



2 

no 


Reach limits 

— 

t flown to 

' 

Singularities 

— 

1 flown to 


Accuracy 

1 

within 12 inches 



2 

more than 12 inches 



1 

within 10 degrees 



2 

more than 10 degrees 


Efficiency 




Time required 

— 

minutes 


Path 

1 

inefficient path 



2 

acceptable path 



3 

efficient path 


CCTV Use 

1 

poor views of payload/ 




orbiter 



2 

acceptable views 




3 

good views 
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Performance Information 


Subject: Date: 


Task 

Berth 

Correct sequence of steps 

i 

yes 

procedures 


2 

no 



If 

no, describe out of step 



actions : 



(refer to PDRS Ops Checklist) 


Safety w 




Vernier w/in 10 ft 

i 

yes 



2 

no 


Reach limits 


# flown to 


Singularities 

t flovn to 


Accuracy 

1 

within 12 inches 



2 

more than 12 inches 


* 

1 

within 10 degrees 



2 

more than 10 degrees 


Efficiency 




Time required 

— 

minutes 


Path 

1 

inefficient path 



2 

acceptable path 



3 

efficient path 


CCTV Use 

1 

poor views of payload/ 




orbiter 



2 

acceptable views 



3 

good views 
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Subject : 


Date : 



Unberth 

procedures 


Correct sequence of steps 


1 yes 

2 no 

If no, describe out of step 
actions : 


(refer to PDRS Ops Checklist) 



Efficiency 

Time required 
Path 




1 within 12 inches 

2 more than 12 inches 

1 within 10 degrees 

2 more than 10 degrees 


minutes 

1 inefficient path 

2 acceptable path 

3 efficient path 


1 poor views of payload/ 
orbiter 

2 acceptable views 

3 good views 
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ITS Survey 


For each of the following questions, circle the number or letter 
that best expresses your opinion. 








1. 

How effectively do you 
content? 

think the 

ITS conveyed the training 


1 

2 3 

4 

5 6 

7 


Not at 
all 




Very 

2. 

To what 
trained 

extent do you 
in the ITS? 

think you have learned the 

tasks 


1 

2 3 

4 

5 6 

7 


Not at 
all 




To a great 
extent 

3. 

How effectively do you 
relating to RMS use? 

think the 

ITS conveyed basic concepts 


1 

2 3 

4 

5 6 

7 


Not at 
all 




Very 

4. 

In terms of using the ITS system, 
task instructions provided by the 

to what extent 
ITS adequate? 

were the 


1 

2 3 

4 

5 6 

7 


Not at 
all 




To a great 
extent 

5. 

How often did you need 
people (e.g., Kalen or 

additional information from other 
Paul) to complete a given ITS task? 


1 

2 3 

4 

5 6 

7 


Not at 




Very 


all 


Training Effectiveness 

70 


To what extent were you able to complete ITS tasks relying 
only on ITS task instructions, the workbook, and the PDRS 
checklist? 


To a great 
extent 


To what extent do you think the ITS is a useful supplement 
to training provided on the higher fidelity simulators? 


To a great 
extent 


To what extent do you think an ITS, in general (given 
adequate task instructions and accurate content), can be an 
effective training supplement to higher fidelity simulators? 


To a great 
extent 


9. Where in the training flow do you think an ITS (similar to 
the one you've been working on) would be most effective? 

a) prior to beginning Level 2000 training on the high 
fidelity simulators 

b) after initial use of the high fidelity simulators 

c) simultaneously with use of the high fidelity simulators 

d) after substantial use of the high fidelity simulators 

e) as a reviewing tool after Level 2000 training is 
complete 


10. To what extent did you 1 ike working on the ITS? 

1 2 3 4 5 6 7 


Not at 
all 


To a great 
extent 
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11. How comfortable did you feel working on the ITS? 

1 2 3 4 5 6 7 

Not at Very 

all 


12. How comfortable would you feel working independently on the 
ITS for short periods of time (i.e., without the presence of 
a trainer)? 

1 2 3 4 5 6 7 


Not at Very 

all 


13. How convenient was it to work on the ITS (e.g., scheduling 
and availability of ITS)? 

1 2 3 4 5 6 7 


Not at Very 

all 


14. How much effort did you invest in trying to successfully 
complete ITS tasks? 

1 2 3 4 5 6 7 


Very 

little 


A great 
deal 


15. To what extent does the ITS encourage you to try to beat the 
system rather than to understand the concepts? 


1 2 3 4 5 6 7 


Not at 
al 1 


To a great 
extent 
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On the following page, please write any additional . comments you 
have on the previous questions (please include question number). 
As well, please discuss any additional issues that were not 
addressed in the questions above. 


